Notes from the crawl frontier
Field notes on crawling and search
Writing about the parts of web crawling that rarely make it into documentation: what breaks at scale, what the standards actually say, and what the web looks like from inside a fetcher.
-
Benchmarking the StormCrawler backends: OpenSearch, URLFrontier and Horizon
The forthcoming Apache StormCrawler® 4.0 adds two utilities for measuring how fast a backend stores URLs and hands them back. Here is what they say about OpenSearch, URLFrontier and Horizon on 168 million status updates from a real crawl, and what they don't say.
Read the post → -
Introducing Ethicrawl
Who we are and where we came from: two decades of web crawling, Apache StormCrawler® and URLFrontier, why the crawling work now has a name of its own, and the values behind it.
Read the post →
More on the way. Crawl scaling,
robots.txt in practice, measured adoption of AI opt-out signals, and why politeness belongs in the frontier. Email contact@ethicrawl.ai to be told when posts go up, or to suggest a topic.