ClaudeBot crawled our content in three bursts and pinged robots and sitemap every two hours in between
In Cloudflare's logs, ClaudeBot spent 80% of 871 requests on /robots.txt and /sitemap.xml and 18% on content. The shape explains it: it swept the test bed in three short bursts (18-20 May, 3-5 July, 1...