AI SEO Projects · Area 1 of 5
Machine-Readable Infrastructure
Last reviewed:
This is the machine-readable layer: the schema, feeds, crawler rules, and page structure that decide whether a model can find your product truth, trust it, and reuse it. It is the least glamorous area and the one every other area depends on. Entity work, content work, and measurement all assume the systems can actually read the page.
Start here, and start with evidence. The first two projects most teams should charter are the bot-policy consolidation and crawler audit and the extractability scorecard — both are cheap audits whose deliverable is a fix list the rest of the program consumes. The infrastructure checklist is the verification pass; the schema templates are the patterns you generate from.
Projects in this area
Bot-policy consolidation & crawler audit
One organization-wide allow/block matrix — per agent, per purpose, per property — reconciled against 90 days of server logs, so policy and reality agree everywhere.
Extractability scorecard & rebuilds
Score the highest-revenue pages on a structure rubric, then rebuild the worst — especially spec tables — into semantic HTML a model can parse, prioritized by revenue times score gap.
Index-freshness pipeline
Wire IndexNow and sitemap pings into the publish path, with special handling for embargoed launches, so content becomes citable in hours instead of days.
Product-feed operations for a hybrid catalog
A feed architecture that natively models hardware, digital goods, and subscription tiers — with per-market truth and launch-state handling — refreshed on AI-surface cadence.
Schema generation pipeline
Generate Product and Offer JSON-LD from the PIM as the single source of truth, with a CI gate that blocks releases when schema and page content disagree.
What this area buys you
- Structured data is the fact source models verify answers against — one controlled study moved GPT-4 from 16% to 54% correct product answers once content was backed by schema.
- Feed and index freshness reach the fastest AI surfaces first — AI shopping surfaces accept refreshes as often as every 15 minutes, and IndexNow pushes feed the Bing index that grounds both ChatGPT search and Copilot.
- Crawler control is now three-dimensional and auditable — you can grant search visibility while declining training, and a log-verified bot policy keeps one template error from silently delisting a market.
Where it goes wrong
- Coverage without accuracy backfires — schema that fails validation or contradicts the visible page creates more problems than no schema at all, and multi-market catalogs multiply the surfaces that can disagree.
- Misconfiguration is silent and replicates — blocking a training bot by default can also block the search indexer you want, and on an enterprise estate the same mistake copies across dozens of domains and CDN configs before anyone notices.
- The vertical's most valuable content is its least parseable — model comprehension drops sharply on irregular HTML and complex tables, exactly the format of hardware spec sheets and edition-comparison grids.
Myths worth striking
- Myth: An llms.txt file improves your AI visibility.
- Reality: Ahrefs analyzed 137K domains and found 97% of llms.txt files received zero requests; Google has publicly declined to support it. Its one real use is developer-docs parsing, not discovery.
- Myth: robots.txt keeps unwanted AI crawlers out.
- Reality: robots.txt is advisory under RFC 9309, not a firewall — non-compliant scrapers ignore it. Access policy and security policy are different jobs and must not be conflated.
- Myth: Allowing an AI company's crawler grants search visibility.
- Reality: Roles are separate. Allowing GPTBot (training) does not grant OAI-SearchBot (search) inclusion; the two are configured independently, and confusing them costs visibility you meant to keep.