Benchmarking the StormCrawler backends: OpenSearch, URLFrontier and Horizon

“Which backend should I use for the status index?” is a fair question to ask about StormCrawler, which ships plenty of options out of the box: OpenSearch, URLFrontier, SQL and Apache Solr among them. In practice most deployments pick OpenSearch, and for good reasons. It is stable: the module is built on the code of the Elasticsearch one, which had to be dropped when the project moved to the Apache Software Foundation for licensing reasons, so it arrived with years of use behind it. It is battle tested, by users who have crawled billions of pages with an OpenSearch cluster in the backend. It does several jobs at once, holding your logs, your crawled content and your frontier in one tool. And it comes with dashboards, so the crawl is something you can watch while it runs.

Every deployment has to put its URLs somewhere. That backend has to accept discovered URLs as fast as the parser emits them, and hand them back to the spouts fast enough to keep the fetchers busy. When it can't, the crawl plateaus or even drops, and from the outside that looks like a slow crawler rather than a slow status index.

A crawl is write-heavy by nature: every page you fetch yields dozens or hundreds of links, nearly all of which have to be offered to the backend, even the ones it has seen a hundred times. On a typical crawl, the StatusUpdaterBolt that writes those URLs to the backend is where you see it first, because it is the component that visibly backs up, and it usually takes the blame.

It only deserves half of it, because the write path is only half of what the backend is doing. Absorbing your URLs and handing them back come out of the same machine, so whichever end is struggling takes resources from the other. And growth has a cost even where it doesn't show up as a slower backend: a bigger store wants more memory, and one given less than its size demands stops answering rather than merely slowing down. That's why you have to judge a backend at the size your crawl will reach, not the size it starts at.

Apache StormCrawler 4.0, which is not out yet but close, adds two utilities to core that measure both halves directly: how fast a status updater bolt writes URLs into a backend, and how fast a spout reads them back out. We used them to compare three backends on the same machine, the same data and the same client configuration, against a snapshot build of that release. Every figure below is the mean of two complete runs of the whole chain, which agreed with each other to within four per cent.

Read figures are from the faster of the two OpenSearch spouts.

Setup

What the utilities measure

StatusUpdaterBenchmark covers the write path. It reads an ndjson dump of URLs, creates status updater bolts from a class name, prepares them with the crawl configuration and feeds them the URLs, then waits for everything to be acked or failed. URLs go straight to the bolt's store method rather than through execute, which leaves the cache of discovered URLs, the scheduler and the metadata filtering out of the measurement. What's left is the bolt and the backend behind it.

SpoutBenchmark covers the read path, against a backend you have already populated. It creates n spouts and m status updaters; every URL a spout emits is handed straight to a status updater as FETCHED, so it is rescheduled and acked back, freeing the slot for the next query. It runs for a fixed number of minutes and reports how many URLs were emitted and how many were unique. Unique URLs per second is the number to watch: a frontier that hands the same URL to ten spouts isn't doing useful work.

The write utility also accepts baseline in place of a real class, a dummy that acks without storing anything. That gives you the ceiling of the harness on your own machine. Use it as a check: if a real backend came anywhere near it, you would be measuring the utility rather than the store. The read utility has no equivalent yet.

The battleground

Everything ran on one laptop, one backend at a time, each in a Docker container with its port published to the host. Swap was off and the page cache dropped before each run.

The input was 168,383,482 status updates from our own Ethicrawl crawler, kept in the order the crawl produced them so that new discoveries and updates to known URLs are interleaved rather than batched. The file is ndjson, the format URLFrontier uses for its own imports and exports, which is what makes a dump like this portable: you can replay the same file into any of the three backends, or into a frontier that has nothing to do with this benchmark.

It contains duplicates, as any real crawl does, since a URL is discovered again by every page that links to it, and we left them all in. The 168.4 million updates collapse into 40.1 million unique URLs over 8.48 million host queues, so three quarters concern a URL the backend already knows about. Recognising URLs it has already seen is most of what a frontier does all day, so a deduplicated input would have measured something a crawl never asks for. Reassuringly, the three backends agreed on the unique total to within a few hundred records, which comes down to slight differences in what each of them considers a valid URL.

OpenSearch was driven through its own module, with opensearch.status.bulkActions: 500, concurrentRequests: 8 and routing on the queue key. URLFrontier and Horizon were driven through the same StormCrawler module and the same settings (org.apache.stormcrawler.urlfrontier.StatusUpdaterBolt, org.apache.stormcrawler.urlfrontier.Spout, urlfrontier.batch.size: 512) because Horizon implements the URLFrontier gRPC API. Between those two, only the server changes.

Results

Writes

Three status updater instances, all 168,383,482 updates, one pass.

The harness is not what any of them ran into. Acking without storing, it managed 3,625,472 URLs/sec on the same machine, about seven times the fastest backend here, so none of these numbers is an artefact of the measuring tool running out of road.

Reads

Three-minute runs against the populated backends, with 10 spout instances, one status updater and topology.max.spout.pending: 10000. OpenSearch was queried with 1,000 buckets of 10 URLs and sampling enabled, which it needs to keep the aggregation affordable at this size. We tested both of its spout implementations.

The read side is where the backends separate most, by a factor of six to eighteen.

What it costs

The runs also recorded what each backend held in memory, what it left on disk and how much processor time it spent per URL, and when you are sizing a machine those count for as much as the throughput. The figures below are the mean of the same two runs, except the OpenSearch read column, which comes from the HybridSpout run. The disk figure for the two frontiers moves by several per cent from run to run, depending on how far RocksDB had got with its compaction when the container stopped, so take it as a size rather than a measurement.

Memory is the one place they are alike. On the read path all three sit between four and seven gigabytes, and giving the JVM backends six gigabytes instead of four changed neither their throughput nor anything else we measured.

The CPU spent per URL, across the backend and the client together, is the honest way to count it when both are on the same machine.

So OpenSearch stores 1.7 times the bytes per URL that Horizon does, spends 2.5 times the processor time to write one and 6.1 times to read one. Reading from URLFrontier took 15.1 of this machine's 20 hardware threads once the client is counted, so on a laptop that number is as likely to be telling you about the host as about the frontier.

Why OpenSearch is slower

Part of each gap is architectural and part of it is how hard the client drives the backend. One status updater instance gets 57,477 URLs/sec out of OpenSearch, less than half of what three manage. The rest is architectural, and it is also why OpenSearch remains a perfectly sensible choice in spite of the tables above.

On the write path

Making a URL searchable costs far more than storing it. Every update has to be turned into index segments and those segments merged, and that work is paid per document, on the write path, whether or not anyone ever queries the field it produced. A frontier only has to remember the URL.

On the read path

The read gap is wider than the problem behind it. An inverted index is a poor match for what a frontier is asked for. “Give me the most urgent URLs, from as many different hosts as possible, no more than a handful per host” becomes an aggregation over the whole status index, run again every few seconds for as long as the crawl lasts. The queues don't exist as objects; they are recomputed from the documents on every query, which is why you need sampling to keep it affordable at 40 million URLs. Sampling and the caps on buckets keep the time per query in hand, but they don't make the aggregation free. Its working set scales with the shard it runs over, so a larger crawl is one you have to give more memory to rather than one you simply wait longer for, and a queue the frontier maintains costs neither.

Aggregations are expensive because of how they are computed. Every document matching the query has to be read, then placed into its bucket. The work is proportional to how much of the index the query touches, which is what the sampling and the caps on buckets and URLs per bucket are there to bound. That bounding is a trade-off in its own right, and a subtler one than the cost it buys: a sampled aggregation looks at part of the index rather than all of it, so what comes back are urgent URLs rather than, strictly, the most urgent ones. Nothing guarantees that the URL most overdue for a fetch is in the batch you get, or in the one after it. For most crawls that is a fine bargain, though you never get asked. A frontier that keeps its queues in priority order gives you that guarantee for nothing, and it goes as soon as the queues have to be recomputed from the documents on every query.

That is also what separates the two spouts, and why both are in the table. The AggregationSpout does everything in one query: a terms aggregation over the status index to build the list of queues, and a top_hits sub-aggregation inside each bucket to pull that queue's URLs. The HybridSpout runs that aggregation far less often, keeps the queues it found, and then issues a small term query for each queue that has run low on URLs. Many cheap, targeted lookups instead of one enormous one, which is where its 5,236 against 2,893 comes from. The trade-off is that it stays on that same set of queues until the next aggregation replaces them. It is quicker because it is looking at less of the frontier, and a queue outside the current set waits its turn however urgent its URLs have become.

The tuning is the second cost, and the one you meet first. A long tail of optimisations went into the Elasticsearch and OpenSearch modules to work around that mismatch: routing by queue key, sampling the aggregation, capping buckets and URLs per bucket, nudging recent dates, bounding query times. Nearly every one arrived as a configuration option. The configuration file we used shows what that adds up to: sixteen settings to make OpenSearch behave, against five for URLFrontier, two of which are a host and a port. None is unreasonable on its own, but together they are a lot to know about a crawl before it has even started.

Trade-offs

What OpenSearch buys you

None of this is an argument against running a crawl on OpenSearch, or on Elasticsearch before it. What the tables measure is one particular use of a search engine: holding a crawl's URLs and handing them back, through StormCrawler modules written to make an index behave like a frontier. Neither system was built for that, and the numbers say nothing about how well either does the job it was built for. The reasons we started with still hold with the tables in front of you, and the first of them is simply that one system covering content, metrics and frontier alike is one thing to deploy, secure and back up instead of three.

The payoff is that everything is queryable. How many URLs are in error on this host, which of yesterday's discoveries are still unfetched, the status breakdown for one path on one site: all of it is a query away, over the frontier and the fetched content alike, which is worth a lot when something has gone wrong.

Nothing on that dashboard needs the frontier to live in OpenSearch, though. The metrics are the crawler's own (pages fetched, URLs waiting in queues, active fetcher threads), emitted by StormCrawler and stored wherever you point its metrics consumer. The frontiers report on themselves as well: URLFrontier has exposed Prometheus metrics since 1.2 and ships a Grafana dashboard and a Loki setup, and Horizon does the same. What you give up by moving the frontier out of the search index is the ad hoc query over individual URLs.

Why URLFrontier is a good choice

URLFrontier implementations decouple the queue from the URLs it holds. A queue is a first-class object with its own state and ordering, so handing out URLs means visiting queues rather than querying URLs. Capabilities that would otherwise be awkward come almost for free: blocking a single queue (a host that asked you to back off, or one that times out on every request) is a flag on one object, not a filter every future query has to carry. Politeness delays and per-queue scheduling work the same way, which is why they belong in a frontier and have to be bolted on elsewhere.

Those capabilities stop being theoretical in StormCrawler 4.0, which puts them to use. A host that answers 429 or 503 with a Retry-After header is taken at its word: we block the queue until the time it asked for. A robots.txt Crawl-delay too long for the fetcher's internal queues used to leave you choosing between erroring the URL out and never crawling the host, or forcing it through at a pace robots.txt didn't sanction. It is now passed to the frontier as a per-queue delay, so the host gets crawled at the rate it asked for. Each is one call against the API, and neither has a natural equivalent where the queues don't exist between queries.

The portability matters more than any figure above. URLFrontier is an API before it is an implementation, and nothing in it is tied to StormCrawler. It's a gRPC service with a published protocol, so the same frontier will serve a crawler written in Python, Go or Rust as readily as one built on Storm, and several crawlers can share one. With a status index in OpenSearch, anything that speaks to OpenSearch can read the documents, but the queueing, the ordering and the politeness live in the crawler's own modules and go nowhere with them. Swapping crawlers becomes a question of which address they point at.

Why you might go for Horizon

Horizon is commercial software with support behind it, from the people who wrote both the specification and the service. It's built so that a crawl is hard to lose: every URL is on disk before it is acknowledged, running out of disk stops the crawl cleanly rather than corrupting it, and an unclean stop restarts in seconds instead of rescanning. It also does things the specification doesn't ask for: queues ranked by the work left in them, whole swathes of a crawl held back by regular expression, crawl-wide counts readable at any size without scanning, signed images with an SBOM and build provenance.

Whether that package is worth a licence is a different question from the one a benchmark can answer, and it's yours to make rather than ours. URLFrontier is open source and free, it's the reference implementation of an API we helped write, and the numbers above are one build of it on one laptop.

What this does and does not tell you

• These are not crawl rates. There is no fetching and no parsing here. A real crawl is bound by politeness delays and the network long before it is bound by the frontier. A backend four times faster won't give you a crawl four times faster; it gives you headroom, and the option of running a bigger crawl on fewer machines.
• These are not the same kind of product. A search engine serving as a frontier is being asked for something it was not shaped for, and it gives you a great deal in return.
• One crawl size. Everything here was measured at roughly 40 million unique URLs. A larger index need not be slower, but it does need more memory. Size OpenSearch's heap for the crawl you are going to have, or the aggregations will start failing instead of just taking longer.
• One machine, two runs. Client and server share the same hardware, and the two runs agreed to within four per cent on every figure, which is tight enough to rank the backends and too loose to read anything into a few per cent. Treat the absolute numbers as specific to this laptop.

Try it yourself

Both utilities live in core in 4.0, take a backend class name and a configuration file, and work with any status updater or spout implementation, including ones not in this comparison. Replace the status updater class name with baseline and you get the ceiling of the harness on your own hardware, which is worth having first: it tells you whether your figures are about the backend or about the tool.

The configurations, the scripts that drove the runs and the instructions for repeating them are all in the benchmark repository, github.com/DigitalPebble/stormcrawler-benchmark.

Wrapping up

If you already run OpenSearch, your crawl is in the tens of millions of URLs and your fetch rate is modest, the status index is very unlikely to be your bottleneck, and the operational familiarity is worth more than the throughput. You also keep a status index you can query, which neither frontier gives you.

Past a few hundred million URLs, or if you want politeness and scheduling to live in the backend rather than in your queries, a purpose-built frontier is the better fit. It's the read side that decides it, because that's the path feeding your fetchers.

Between the two frontiers, on this hardware, Horizon wrote 2.3 times faster and read 1.5 times faster than URLFrontier. Both speak the same API through the same StormCrawler module, so swapping one for the other is a configuration change, which means you can run this comparison on your own data rather than taking ours.

Thanks to everyone who has contributed to StormCrawler and URLFrontier over the years, and to the users who kept asking the question that led to these utilities being written. We'd love to see numbers for the backends we didn't cover, SQL and Apache Solr among them: same utilities, same method, different class name. If you run them, do tell us!

Happy crawling!