Technical GEO: AI crawlers, robots.txt, llms.txt and structured data

Know the AI crawlers

AI companies use several user agents, and they do not all do the same job. Some collect training data, others fetch pages to answer a user in real time. The main ones documented by their operators:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results), ChatGPT-User (pages fetched on a user's request).
• Perplexity: PerplexityBot (search index) and Perplexity-User (fetches on a user's request).
• Anthropic: ClaudeBot and related Claude user agents.
• Google: Googlebot powers Search, including AI Overviews. Google-Extended is a separate control for use in Gemini models and does not affect Search.
• Operators update these lists, so check each company's crawler documentation before you edit your rules.

Check robots.txt and your firewall

Open yoursite.com/robots.txt and look for rules that block these user agents, or a blanket Disallow. Blocking training crawlers is a legitimate choice; blocking the search and user-request agents usually removes you from live answers.

Also check your CDN or firewall. Bot protection often blocks AI user agents before robots.txt is even read. If an assistant cannot fetch your page when a user asks about you, it will describe you from other sites instead.

llms.txt: useful, but not a switch

llms.txt is a proposed convention: a Markdown file at the root of your site that summarises what you do and links to your most useful pages, written for language models. It is cheap to add and helps any agent that reads it.

Be clear about its status. It is a proposal, not a standard that the major assistants have committed to follow, so do not expect it alone to change how often you are named.

Structured data that describes you precisely

Schema.org markup gives machines unambiguous facts about your pages. It is not a guaranteed ranking factor for assistants, but it removes guesswork about who you are and what you sell.

• Organization on the home page: name, logo, URL, social profiles.
• Product or Service on offer pages, with a clear description.
• FAQPage where the page really has questions and answers.
• Article on guides and posts, with author and dates.
• Only mark up what is visible on the page.

Make the page readable without JavaScript tricks

Many crawlers do not run JavaScript the way a browser does. Put the key facts in the HTML: what you do, for whom, pricing model, locations. Avoid putting important text only in images, and keep pages fast. Server rendering or static pages are the safe choice.