372 of this host's 3,522 daily “unique visitors” were one crawler: Amazonbot arrived on 372 addresses under 1 user-agent string, from one network — AI Crawler Index
www.pathwren.workers.dev
A dated measurement report from one small public host. In the 24 hours to 2026-09-06T12:50:45Z: 1,161 of 3,522 client keys (33.0%) were announced crawlers, folding them to (declared name, ASN) removes 1,003 of them and takes the headline to 2,519 — and the same rule refuses to fold Mastodon, which is 230 keys across 89 networks. Every figure with its derivation and the SQL. CC0.

A dated measurement from one small public host, with the method written out so you can check it or argue with it.

In the 24 hours to 2026-09-06T12:50:45Z these logs held 3,522 distinct client keys. 372 of them were one crawler.

  • Amazonbot: 881 requests, 372 addresses, one user-agent string, every address in AS14618. 323 of those addresses had been seen here before the window opened, so it is a stable pool being re-keyed, not churn and not a botnet. 54 fresh addresses inside one 4-minute burst.
  • Fold every announced crawler to (declared name, ASN) and 1,003 of the 3,522 keys disappear: the headline drops to 2,519, 28.5% lower, without discarding a single request.
  • The same rule refuses to fold Mastodon, which is 230 keys across 89 networks. That is real fan-out from 89 instances and folding it to one would be a lie - the rule has to be able to say no, or it is just a smaller number.
  • The opposite shape, same window: ClaudeBot, 3,916 requests and 16.2% of all rows, from ONE address and one client key. So “unique visitors” over-counts and under-counts on the same day, in opposite directions, and neither error is visible in the total.

Posted because the fix is not a vendor’s dashboard: it is a definition. Every number in the report has its derivation in a CC0 figures file next to it, including what counts as a client key and what counts as an announced crawler, so you can disagree with the definition rather than with me. The index it leans on covers 150 crawlers from 74 operators, with the published IP ranges mirrored for exactly the check this report runs.

https://www.pathwren.workers.dev/c/lemmy/blog/crawler-fleet-fold-2026-w36.html

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)