We wrote the crawlers other people run

Open-source crawling, from the inside

Our expertise is not a claim on a slide. It is in public repositories, mailing-list archives and release notes going back years.

• Apache StormCrawler®Original author, committer & PMC member
  Started in 2013 as a collection of resources for building low-latency, customisable and scalable crawlers on Apache Storm®. It became the default choice for teams who needed continuous crawling rather than batch, and graduated to a top-level Apache Software Foundation project in June 2025.
• URLFrontierCreator & maintainer
  A crawler-agnostic API and service for crawl frontiers, built so that frontiers and fetchers can evolve independently, with politeness, prioritisation and crawl scheduling expressed in the protocol itself rather than buried in each crawler.
• Apache Nutch®Former committer & PMC chair
  Years of work on the original open-source web crawler (plugins, filters and releases) and years of running it for clients at a scale of 10⁹ pages, which is where the hard-won operational knowledge comes from.
• Crawler-CommonsContributor
  The shared library behind many crawlers for robots.txt parsing, sitemap handling and URL normalisation, the unglamorous details that decide whether a crawler is well behaved or not.
• Apache Software FoundationMember
  Elected member of the foundation, with involvement across several projects in the crawling and processing ecosystem, including Apache Storm®.

Crawling at web scale, for real clients

Over the years our crawling and search work has been deployed for organisations ranging from stealth-mode startups to NASDAQ-listed companies, in business intelligence, media monitoring, e-commerce, recruitment, telecommunications and public research.

That has included web-scale crawl infrastructure for public research datasets, low-latency vertical crawls refreshed continuously, broad multilingual crawls of the European web, and more than one rescue of a crawl that had stopped scaling.

Published references are a matter for each client to agree to, so this page names the code rather than the logos. We are happy to discuss comparable work, in confidence, when we talk.

Why we work this way

A crawler is a guest on someone else's infrastructure. Every request costs the host bandwidth, CPU and, ultimately, electricity. Treating that as free is how the current mess came about: publishers blocking whole ranges, cat-and-mouse games with proxies, and a collapse of trust that makes legitimate crawling harder for everyone.

Doing it properly turns out to be the pragmatic choice as well as the decent one. Crawlers that identify themselves, respect limits and honour opt-outs do not get blocked, do not need an ever-growing proxy budget, and do not become a liability. That is the standard we build to, and it is not negotiable in our work.