Robots (SEO & GEO)
The platform includes configurable Search Engine Optimization (SEO) and Generative Engine Optimization (GEO) controls. SEO manages how traditional search engines like Google and Bing index your site. GEO controls how AI systems like ChatGPT, Claude, and Gemini discover and use your content.
Robot controls are split into two independent sections: Read Access governs who may see your content (crawling, indexing, AI training), while Write Access governs who may act on your site (placing orders, submitting forms, uploading files). The two are never coupled — changing AI read access does not open or close checkout to AI assistants, and vice versa.
Read Access
Read Access controls who may crawl, index, and train on your content. Search engine settings use standard directives from the Robots Exclusion Protocol, honored by all major search engines; they control the meta robots tag and the /robots.txt file. AI settings additionally cover training and image use.
| Setting | Standard | Default | Effect |
|---|---|---|---|
| Allow Indexing | Robots Exclusion Protocol | On | When off, search engines will not add your pages to their results (noindex) |
| Allow Link Following | Robots Exclusion Protocol | On | When off, search engines will not follow links on your pages (nofollow) |
| Allow AI Crawlers | Robots Exclusion Protocol | On | Master toggle for AI bots reading your content. When off, all known AI bots are blocked via robots.txt and the page opts out with noai + noimageai — your business stops appearing in answers from ChatGPT, Claude, Perplexity, and Google AI Overviews |
| Allow AI Indexing | Experimental | On | When off, adds noai meta tag to opt out of AI training and retrieval. Not all providers honor this. |
| Allow AI Image Use | Experimental | On | When off, adds noimageai meta tag to opt out of AI image training. Not all providers honor this. |
Write Access
Write Access controls who may take actions on your site — placing orders, creating accounts, submitting contact forms, uploading files. Bot detection on these endpoints is provided by our bot-protection partner, Vercel. Unknown bots and scripted traffic are always blocked from write actions, regardless of any setting here.
| Setting | Default | Effect |
|---|---|---|
| Allow AI Assistants | On | Some customers let an AI assistant (like ChatGPT) act for them. When on, assistants that Vercel has verified are working for a real person may place orders, sign up, and submit forms. When off, every action requires a person in a browser — safer against automation, but genuine assistant-driven orders and signups are turned away too. |
Blocked AI Bots
Block specific AI bots by user-agent name, even when the Read and Write toggles above would otherwise allow them. Each entry has two flags, so you can block one side of a bot's behavior while allowing the other:
| Field | Default | Effect |
|---|---|---|
| Bot User-Agent | — | The bot to block, chosen from the known AI bots list below |
| Block Reading | On | Blocks the bot from crawling and indexing your content (per-agent robots.txt rule) |
| Block Writing | On | Blocks the bot from taking actions — orders, accounts, forms, uploads — even when it is a verified assistant and Write Access would otherwise allow it |
A newly added bot is blocked from both reading and writing. Uncheck one flag to carve out that side — for example, blocked from AI training but still allowed to check out on behalf of a customer. An enabled block here beats every other setting in the Robots section.
Block Writing covers a bot's whole product family, not just the one name. Blocking ChatGPT-User also blocks OpenAI's Operator agents and GPT Actions from acting on your site — OpenAI files all of that traffic under the one user-agent token, even though the verification layer sees them as separate identities.
anthropic-ai, FacebookBot, Applebot-Extended are robots.txt-only names — nothing ever acts on a site under them, so there is nothing for a write block to catch. Scripted traffic impersonating them is rejected as unverified automation regardless.Enforcement Layers
Robot controls are enforced at multiple levels for maximum coverage:
- robots.txt — Dynamic, database-driven rules. Primary read-side enforcement for well-behaved crawlers.
- Meta robots tags —
index/followplus thenoaiandnoimageaidirectives in the HTML<meta>tag. Database-driven, and the layer that reaches an AI bot which ignores robots.txt. - Bot verification on write endpoints — Vercel bot detection challenges every order, signup, form submission, and upload. Enforces the Write Access toggle and per-bot Block Writing flags; unverified automation is always rejected.
- llms.txt — Structured site description endpoint that AI systems can consume.
Known AI Bots
The platform recognizes these AI bot user-agents for per-agent robots.txt blocking. Note that most vendors run more than one crawler: a training bot that collects content to build models, and a separate search or user-initiated bot that fetches pages at the moment someone asks a question. Blocking only the training bot leaves your content readable by the assistant itself, which is why both appear below.
| Bot | Organization | Type | What it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Trains OpenAI foundation models |
| OAI-SearchBot | OpenAI | Search index | Indexes for ChatGPT search — NOT covered by blocking GPTBot |
| ChatGPT-User | OpenAI | User-initiated fetch | Fetches a page when a ChatGPT user asks about it (OpenAI says robots.txt may not apply) |
| ClaudeBot | Anthropic | Training | Trains Anthropic models |
| Claude-SearchBot | Anthropic | Search index | Indexes for Claude's search — NOT covered by blocking ClaudeBot |
| Claude-User | Anthropic | User-initiated fetch | Fetches a page when a Claude user asks about it |
| anthropic-ai | Anthropic | Training | Legacy token, kept so older saved config keeps working |
| Google-Extended | Training | Gemini training and grounding only — NOT Google Search, NOT AI Overviews | |
| Google-CloudVertexBot | Training | Crawls sites for Vertex AI grounding | |
| PerplexityBot | Perplexity | Search index | Indexes for Perplexity answers |
| Perplexity-User | Perplexity | User-initiated fetch | Fetches a page when a Perplexity user asks about it — Perplexity says it generally IGNORES robots.txt |
| Meta-ExternalAgent | Meta | Training | Trains Meta AI models |
| Meta-WebIndexer | Meta | Search index | Builds Meta AI's search index |
| Meta-ExternalFetcher | Meta | User-initiated fetch | Fetches a page for a Meta AI request — Meta says it MAY bypass robots.txt |
| FacebookBot | Meta | Training | Legacy Meta crawler token |
| Applebot-Extended | Apple | Training | Apple Intelligence training — does NOT affect Siri or Spotlight search |
| CCBot | Common Crawl | Open dataset | Open dataset that many other models train on |
| Bytespider | ByteDance | Training | ByteDance / TikTok model training |
| Amazonbot | Amazon | Training | Alexa and Amazon AI services |
Google-Extended only controls Gemini AI training and grounding. Blocking it does not affect your site's Google Search rankings or indexing — and it does not remove you from Google AI Overviews, which are generated from Google's ordinary search index. The only setting that changes what AI Overviews can use is Allow Indexing, and turning that off removes you from Google Search entirely.ChatGPT-User, Perplexity says Perplexity-User "generally ignores" them, and Meta says Meta-ExternalFetcher "may bypass" them. Anthropic documents Claude-User as honoring them. So blocking AI crawlers reliably stops your content being ingested and indexed; it does not guarantee nobody can pull up your page through an assistant.llms.txt
The /llms.txt endpoint provides a structured text description of your site that AI systems can read. This is an emerging standard (see llmstxt.org) that helps AI chatbots accurately answer questions about your business. Publishing is controlled by the Publish llms.txt toggle (off by default); when enabled, the llms.txt Content field seeds the endpoint.
If you leave the llms.txt content field empty, the platform auto-generates content from your site name, description, location, and contact information. For best results, write custom content that includes:
- A clear description of what your business does
- Your location and service area
- Products and services offered
- Contact information
- FAQ section answering common customer questions
