The opt-out protocols we honour

robots.txt

The foundation of crawler etiquette since 1994, now standardised as RFC 9309. A site’s robots.txt tells crawlers which parts of the site they may visit, per user agent.

If robots.txt disallows our crawler — by name or via the * wildcard — the site never enters the corpus. See the crawler page for how we identify ourselves and more blocking examples.

We go beyond our own name. Most sites that object to AI crawling have never heard of ethicrawl — their robots.txt blocks the crawlers they do know about. So we treat directives aimed at any well-known AI training crawler as if they were addressed to us:

The names we obey cover the AI training crawlers and training-control tokens of OpenAI, Google, Apple, Meta, Anthropic, ByteDance, Cohere, Common Crawl and others — blocking any one of them keeps the site out of our corpus. Directives aimed at AI search or assistant fetchers are not interpreted this way, as blocking those does not express a training opt-out.

Content Signals

Content Signals extend robots.txt with a declaration of how content may be used, not just whether it may be fetched. A single line expresses separate preferences for search, AI answers and AI training:

We parse the signal that applies to our user agent group. If it declares ai-train=no, the site is excluded from the training corpus — even when plain crawling is allowed.

TDM Reservation Protocol

The TDM Reservation Protocol (TDMRep) is a W3C Community Group specification that lets rightsholders reserve their text-and-data-mining rights, as provided for by the EU Copyright Directive. A reservation can be declared through an HTTP header:

or site-wide through a /.well-known/tdmrep.json file:

We check both — per the specification, header values take precedence over the file. A reservation on the site root keeps the site out of the corpus, and any declared policy URL is recorded alongside it.

noai directives

The noai and noimageai directives, first introduced by DeviantArt, ask that content not be used for AI training. They are usually deployed as a server-wide X-Robots-Tag response header:

We inspect the headers returned with a site’s robots.txt: a noai directive excludes the site from the corpus. noimageai is recorded but has no further effect, as we do not collect images.

Signals we record

Two further signals are collected purely to study their adoption — they express licensing terms or guidance rather than an opt-out, so they do not affect whether a site enters the corpus:

Really Simple Licensing (RSL) declares machine-readable licensing terms through a License directive in robots.txt pointing to an XML licence file. We record the declaration and archive the licence file for analysis.

llms.txt is a curated, LLM-friendly map of a site — a welcome mat rather than an opt-out. We record its presence and archive its content.

How we apply them

Every hostname we consider goes through the same checks before a single content page is fetched:

• ✓robots.txt disallows us — or any well-known AI training crawler? The site is not crawled at all.
• ✓A Content Signal declares ai-train=no? The site is excluded from the corpus.
• ✓A TDM reservation applies to the site? The site is excluded from the corpus.
• ✓An X-Robots-Tag header declares noai? The site is excluded from the corpus.
• ✓Every decision is recorded with its reason, so exclusions are auditable and permanent until the site changes its preferences.

Questions?

Unsure how to declare your preferences, or want to check what we hold for your domain? Contact us directly.