ClaudeBot ate my website and then disappeared.

Conceptio · October 2026 · the crawler logs, read end to end

I launched Conceptio quietly in early September as an open-access search engine for papers, standards and case law. With no announcement, no backlinks, and the site asking not to be indexed, on September 9 the domain went live and the TLS certificate was issued. The next day ClaudeBot made 10,884 requests.

Nobody had linked to the site anywhere. My best guess is certificate transparency: a new TLS cert is a public announcement that a domain exists, and something in Anthropic's discovery pipeline reads those logs. Within 24 hours of the only public signal I had emitted, the crawling began.

It never really stopped for four weeks. ClaudeBot ramped from ~10k requests a day to a peak of 143,475 on September 23 (one crawler, one day, on a site with no users to speak of). Over the whole month it made 968,936 requests. The site is small and the corpus behind it is not, so I watched the logs the way you'd watch weather.

• ClaudeBot
• GPTBot
• Amazonbot
• Googlebot
• OAI-SearchBot
• ChatGPT-User

GPTBot had its own schedule: a five-day bulk run from September 29 to October 3 totaling 421,522 requests, peaking at 134,571 in a single day, and then it too went quiet. Amazonbot arrived October 2 and settled at a flat ~20k requests a day, which is the most polite behavior in this entire dataset.

Then came October 4. The site was finally ready to be found, so I flipped the noindex directive off, removing the one line telling crawlers to keep out. ClaudeBot's last meaningful day was October 3. Its final 32 requests landed on October 4, and then nothing. GPTBot, already done with its bulk run, never came back.

Starting that same day, OAI-SearchBot woke up. Five requests on October 4, then 49, 127, 159, 410, 631 by October 9, tracing a curve that doubles every couple of days. ChatGPT-User, the fetcher that backs links inside ChatGPT, follows behind it: 203 requests on October 9. Read the two panels of the chart together and the handoff is hard to miss: the training crawlers ate an unindexed site for a month, and the moment it became findable, the search layer took over the table.

What I find interesting is not that AI companies crawl the web, since that part is well known. Rather, it is the shape of the handoff. A noindex directive is a request, and for four weeks the only visitors I had ignored it entirely. The day it came off, the crawlers that train on the web left within hours, and the crawlers that retrieve from it appeared the same day. Discovery was instant in both directions.

Two footnotes for the curious. First, the data: the complete Cloudflare crawler export for this period is published with this post at crawlers.csv, unmodified, including the days that don't flatter anyone. Second, the corpus itself is open as one JSONL dump, public by design. The documents behind Conceptio belong to the commons their authors put them in; gatekeeping data that isn't ours doesn't sit right.

Conceptio is one sentence: an open-access search engine for papers, standards and case law, with licenses and provenance on every hit. If you build with documents, it's at conceptio.app. The crawlers already know.