What ethicrawl does, and how to block it

What it does

ethicrawl is the crawler behind Ethicrawl.ai. It fetches publicly available web pages to build multilingual training corpora for AI research, with a particular focus on Europe’s languages — EU and beyond.

Every crawl respects robots.txt, applies polite request delays, avoids IP farms or scraping shortcuts, and runs on infrastructure hosted in clean-electricity regions.

How it identifies itself

We don’t hide behind anonymous or spoofed user agents. Every request from our crawler carries this User-Agent string:

If a request claims to be ethicrawl but doesn’t match this string, it isn’t us.

How to block it

To stop ethicrawl from crawling your entire site, add this to your robots.txt:

To block specific paths instead of the whole site:

We check robots.txt before every crawl of a domain, and honour Disallow and Crawl-delay directives immediately — no grace period, no exceptions.

Already blocking other AI crawlers? That’s enough. We go the extra mile: directives aimed at well-known AI training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and others — are treated as applying to us as well. If your robots.txt already tells any of them to stay away, ethicrawl stays away too, without you ever having to know we exist.

Prefer to keep your site searchable but out of AI training? We also honour Content Signals and the TDM Reservation Protocol.

What you can expect from us

• ✓ Respects robots.txt on every domain, every time
• ✓ Transparent, non-anonymous User-Agent string on every request
• ✓ Real politeness delays between requests, no hammering your servers
• ✓ No IP farms, no scraping shortcuts, no spoofed identities

Questions?

For anything not covered here — crawl scope, opt-outs, or specific domains — contact us directly.