From the first fetch to the search results

Crawler design & development

Whether you need a focused crawl of a few thousand sites refreshed every hour, or a broad crawl of a sizeable slice of the web, the design decisions made in the first week determine what the system costs to run for years.

We start from what you actually need to collect, then choose the architecture to match, rather than the other way round.

• ✓ Technology selection: Apache StormCrawler®, Apache Nutch®, URLFrontier, or something smaller and simpler if that is genuinely enough
• ✓ Frontier and scheduling design: discovery, prioritisation, recrawl intervals, politeness budgets
• ✓ Custom parsers, extractors and URL filters for the sites and formats you care about
• ✓ Sitemap, feed and API ingestion alongside plain crawling, since the cheapest page is the one you did not have to discover
• ✓ Handling of JavaScript-heavy sites, where and only where it is worth the cost

Scaling, tuning & operations

Crawls rarely fail outright. They plateau: the fetch rate stops climbing, the frontier grows faster than it drains, the cloud bill grows faster than the dataset. These are our favourite problems.

• ✓ Diagnosis of throughput ceilings: DNS, connection handling, queue skew, politeness, or simply too few threads in the wrong place
• ✓ Cluster sizing and cost reduction, including the energy footprint of the crawl
• ✓ Metrics, dashboards and alerting that show what a crawl is really doing, not just that it is alive
• ✓ Deployment and operations: containers, orchestration, and running the thing day to day if you would rather we did
• ✓ Incident work: crawls that are blocked, throttled, looping, or quietly collecting rubbish

Search & relevance

A crawl is only as useful as what sits on top of it. We have been building retrieval systems for as long as we have been building crawlers, and the two are rarely designed well in isolation.

• ✓ Index and mapping design on OpenSearch, Elasticsearch or Apache Solr®, sized for your query load
• ✓ Ingestion pipelines from crawl to index, with the enrichment done once and in the right place
• ✓ Relevance tuning backed by evaluation, not opinion
• ✓ Hybrid lexical and vector retrieval, and retrieval for RAG pipelines, where it earns its complexity
• ✓ Query performance and cluster sizing for clusters that have outgrown their design

Web data pipelines & datasets

Raw pages are not a dataset. Getting from one to the other is where most web data projects lose time, and where the quality of the final product is decided.

• ✓ Content extraction, boilerplate removal and conversion to text, Markdown or structured records
• ✓ Language identification, deduplication, quality filtering and classification
• ✓ Large-scale batch processing with Apache Spark® or Apache Flink®
• ✓ Storage and interchange formats that stay usable: WARC for provenance, Parquet for analysis
• ✓ Corpus curation for AI teams, with documented provenance and opt-out handling throughout

Training, mentoring & support

Most clients want to end up independent. We are happy to be the reason they get there.

• ✓ Workshops on web crawling at scale, frontier design, and search engineering, tailored to your stack
• ✓ Architecture review and code review for teams building their own crawlers
• ✓ Retained support: a known number of days per month, and someone who already understands your system when it breaks
• ✓ Second-line escalation for teams running open-source crawlers in production

How we work

Small, senior, hands-on. You get the people who wrote the software, not a layer of account management on top of them.

Open source at the core. We build on projects we maintain and contribute our improvements back, and we leave your team able to run what we deliver. Where our own tooling is the better answer we will say so, and we are clear from the outset about what is open source and what is licensed.

Honest scoping. If your problem is smaller than you fear, we will say so. If it is bigger, you will hear that before the invoices start.

Politeness is not optional. We will not build crawlers that disguise their identity, rotate through residential proxies, or ignore an opt-out. It is the one thing we turn work down over, and the reason our crawls keep working.