The Ethicrawl crawler, and how to block it

What it does

The Ethicrawl crawler is ours. It is not a product, and it does not crawl on behalf of clients. We run it for our own research projects, and it fetches publicly available web pages only for those.

That research covers two things at the moment: building multilingual text corpora for Europe’s languages, including the regional and minority languages that large datasets usually miss; and measuring how sites declare their crawling and AI-usage preferences across the web, so that the state of those protocols is something we can report on from data rather than anecdote.

Crawling the web is also what we do for a living, and the crawlers we build for clients are held to the same rules as this one. Our crawler is where we prove they work.

Every crawl respects robots.txt, applies polite request delays, avoids IP farms or scraping shortcuts, and runs on infrastructure hosted in clean-electricity regions.

How it identifies itself

We don’t hide behind anonymous or spoofed user agents. Every request from our crawler carries this User-Agent string:

If a request claims to be our crawler but doesn’t match this string, it isn’t us.

A user agent is easy to fake, so we also publish the IP addresses our crawler sends its requests from, in crawler-ips.json. The file uses the same format as the lists Google and Bing publish for their own crawlers, which means tools that already verify those can read ours without any changes. creationTime tells you when the list was last generated, and each entry under prefixes is an address range in CIDR notation. A request carrying our user agent from an address outside these ranges isn’t from us, and you can block it without affecting our crawler.

How to block it

To stop our crawler from visiting your site at all, add this to your robots.txt:

To block specific paths instead of the whole site:

We check robots.txt before every crawl of a domain, and honour Disallow and Crawl-delay directives immediately, with no grace period and no exceptions.

Already blocking other AI crawlers? That’s enough. We go beyond our own name: directives aimed at well-known AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and others) are treated as applying to us as well. If your robots.txt already tells any of them to stay away, our crawler stays away too, without you ever having to know we exist.

Prefer to keep your site searchable but out of AI training? We also honour Content Signals and the TDM Reservation Protocol.

What you can expect from us

• ✓ Respects robots.txt on every domain, every time
• ✓ Transparent, non-anonymous User-Agent string on every request
• ✓ Real politeness delays between requests, no hammering your servers
• ✓ No IP farms, no scraping shortcuts, no spoofed identities

Questions?

For anything not covered here (crawl scope, opt-outs, or specific domains) contact us directly.