Horizon, a frontier your crawl can grow into
Horizon is a URL frontier service by DigitalPebble: one process that remembers every URL your crawler has found, decides what it should fetch next, and keeps it polite to the sites it visits. Fast enough to stay ahead of a crawler fetching flat out, and built so that a crawl is hard to lose.
It is an implementation of URLFrontier, the open API we created for exactly this job, so Apache StormCrawler® works with it out of the box, and so does any other crawler that speaks the protocol.
What it does
The frontier is the memory of a crawl. It holds the URLs that have been discovered, those already fetched, those due to be looked at again, and the order and timing that keep each host from being hammered. Get it wrong and a crawl plateaus, loses its place on a restart, or quietly turns into a problem for the sites it depends on.
Horizon does that job as a service of its own, built for crawls that run for months and only get bigger. It stays ahead of a crawler fetching at full tilt, and it does not lose your crawl. Every URL is on disk before it is acknowledged, a crash costs you nothing that was acknowledged, and a restart picks the crawl up where it left off.
Politeness delays, per-queue scheduling and recrawl timing live here, once, instead of being reimplemented in every crawler you run.
It works with the crawler you already have
URLFrontier is an open, gRPC-based API, developed in the open at Crawler-Commons so that crawlers and frontiers can evolve independently of each other. Horizon implements that API unchanged. It adds nothing you have to adopt and nothing you have to work around. What it offers beyond the specification is there when you want it, and out of the way when you do not.
- ✓ StormCrawler® Use the URLFrontier module it already ships with, point it at Horizon, and run. No code to write, no fork to maintain.
- ✓ Any other crawler It is gRPC, so a client comes out of the published protocol definition in Java, Python, Go, Rust, C#, Node and the rest, whether your crawler is off the shelf or your own.
- ✓ Already on URLFrontier Horizon is a drop-in replacement for another implementation. Change the address, keep the crawler.
- ✓ No lock-in Everything your crawler asks of Horizon is in the open specification, so you can move to a different implementation whenever you want to. That is the whole point of having a specification.
What you get
- ✓ Throughput A single machine keeps up with a crawler running flat out, with no cluster to operate: one container, one data volume.
- ✓ Robustness Nothing acknowledged is lost to a crash, the disk is guarded so a crawl stops cleanly rather than corrupting itself, and an unclean stop restarts in seconds instead of rescanning the crawl.
- ✓ Beyond the standard Queues ranked by the work left in them, whole swathes of a crawl held back by regular expression, and crawl-wide counts you can read at any size without scanning anything.
- ✓ Visibility Prometheus metrics and a Grafana dashboard that show what the frontier is really doing, queue by queue.
- ✓ Supply chain A signed container image for amd64 and arm64, with an SBOM and build provenance.
- ✓ Support From the people who wrote both the specification and the service.
If you would rather not design the crawl around it yourself, that is what we do for a living.
Licensing
Horizon is commercial software, licensed per deployment. We are straightforward about that: the specification it implements is open, the service itself is ours.
Evaluation licences are available on request: try Horizon against your own crawler, on your own data, before committing to anything.
The licence is checked entirely offline. Nothing about your crawl, your URLs or your deployment is ever sent over the network. There is no telemetry and no call home.
Interested?
Tell us roughly what you are crawling (how many sites, how many URLs, how often) and we will tell you straight whether Horizon is the right answer for it.
Ask about Horizon →