AI SEO Projects ·Machine-Readable Infrastructure
Bot-policy consolidation & crawler audit
Last reviewed:
- Owner
- Technical SEO, with security/WAF sign-off
- Metric
- Share of properties serving the central policy, with drift monitoring green
- Clock
- Retrieval layer —the two clocks
Crawler control is three-dimensional: OpenAI alone documents GPTBot (training), OAI-SearchBot (search visibility), and ChatGPT-User (user-triggered fetches). You can allow search while declining training — but only if the policy is deliberate and consistent. This project builds one allow/block matrix, per agent and purpose and property, then proves it against reality with logs.
Why this, mechanically. A default block on a training bot can also block the search indexer you actually want, and on an enterprise estate that mistake replicates across dozens of market domains and CDN configs before anyone sees it. Overzealous WAF rules do the same damage from the other direction, blocking legitimate AI bots the security team never meant to. The matrix plus a log audit is what turns “we think our policy is right” into “we verified it.”
Deliverable. A versioned, owned bot-access matrix deployed to every property; a 90-day log study segmenting GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot; a remediation list for divergences and WAF false-positives; and drift monitoring.
Finish line. Every property serves the same policy, logs confirm it, and a drift check watches for regressions.
Background: How AI agents fetch pages and ChatGPT search vs GPTBot vs OAI-SearchBot. Procedures: enable AI search access and parse logs for AI bot behavior.