Where is the robots.txt file located?

It must sit at the root of the host, such as www.example.com/robots.txt. Crawlers don't look for it in subfolders, and each subdomain needs its own file, so blog.example.com and app.example.com are handled separately. Webflow, Framer and most hosts let you edit it in site settings, and Next.js can generate it from a robots file in the app directory.

Does robots.txt stop a page from appearing in Google?

Not reliably. Robots.txt stops Google from crawling a URL, but if other pages link to it, Google can still index the URL and show it without a description. To keep a page out of results, let Google crawl it and add a noindex meta tag or header, or remove the page and return a 404 or 410.

Do all bots follow robots.txt?

No. Major search engines and the main AI companies say their crawlers honor it, but the protocol is voluntary. Scrapers and malicious bots can ignore it entirely. If you need to stop a specific bot that ignores robots.txt, block it at the server or through your CDN's bot management, such as Cloudflare's controls.

How do I test my robots.txt file?

Open the file in a browser first and read it line by line. Then use the robots.txt report in Google Search Console, which shows the version Google fetched and any errors. Use the URL Inspection tool to check whether a specific page is blocked. Retest after every launch, platform change or edit.

All posts

SEO & AI search

What Is robots.txt? A Plain-English Definition

What is robots.txt? A plain definition, a robots.txt example with AI crawler lines for GPTBot, OAI-SearchBot and PerplexityBot, and the mistakes that block SEO.

By Mehdi HassanOct 29, 20264 min read

The short answer

Robots.txt is a plain text file at the root of a website, such as example.com/robots.txt, that tells crawlers which parts of the site they may or may not request. Search engines and most AI crawlers follow it voluntarily. It controls crawling, not indexing, so it's the wrong tool for hiding a page from Google.

Key takeaways

Robots.txt tells crawlers where they may go. It's a request that well-behaved bots honor, not security.
Blocking a URL in robots.txt stops crawling, but the URL can still appear in results. Use noindex for that.
AI companies use separate crawlers for training and for search, so you can allow one and block the other.
One stray Disallow line can remove a whole site from search. Check robots.txt after every launch.

Part of Web Design, SEO and AEO Glossary: 50 Terms Explained Simply

What is robots.txt?

Robots.txt is the first file many crawlers request when they visit a site. It follows the Robots Exclusion Protocol: a set of rules grouped by user agent, the name a crawler uses to identify itself, with Allow and Disallow lines listing paths. It can also point crawlers to your XML sitemap.

It matters because it's a single file with site-wide reach. A good one keeps crawlers out of admin areas and endless filter URLs. A bad one can hide your whole site. Google's introduction to robots.txt covers the syntax, and the related terms sit in our SEO and AEO glossary.

What does a robots.txt example look like?

A robots.txt file is a few short groups of lines. The example below is a sensible default for a B2B marketing site that wants to appear in search and AI answers but keep its app and internal pages out.

Example robots.txt with AI crawler lines (adapt the paths to your site)

User-agent: * Disallow: /app/ Disallow: /search

User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: GPTBot Disallow: /

Sitemap: https://www.example.com/sitemap.xml

The first group applies to every crawler and keeps them out of the product app and internal search. The next two groups explicitly allow the crawlers behind ChatGPT search and Perplexity answers. The GPTBot group blocks OpenAI's training crawler, a choice some companies make and others don't. The last line points every crawler to the sitemap.

Whether to block training crawlers is a business decision, not an SEO one. We set out both sides in should you block or allow AI crawlers.

Which AI crawlers can robots.txt control?

Robots.txt can control most major AI crawlers, because the companies behind them publish their user agent names and say they honor the file. The key detail is that training and search often use different bots.

User agent	Operator	Used for
Googlebot	Google	Search, including AI Overviews and AI Mode
Google-Extended	Google	A control for Gemini model training, not Search
OAI-SearchBot	OpenAI	Showing sites in ChatGPT search
GPTBot	OpenAI	Collecting data for model training
ClaudeBot	Anthropic	Collecting data for model training
PerplexityBot	Perplexity	Indexing pages for Perplexity answers

Blocking Google-Extended doesn't remove you from AI Overviews, because those run on Googlebot's crawl. Check each operator's documentation for current names, since they change.

What are the most common robots.txt mistakes?

The most common robots.txt mistakes are blocking the whole site by accident and using Disallow to hide pages that should be noindexed.

Blocking pages you want deindexed. If Google can't crawl a page, it can't see a noindex tag on it. Allow crawling and add noindex instead.
Blocking CSS and JavaScript. Google needs them to render pages. Blocking them can make pages look broken to the crawler.
Treating it as security. The file is public and only polite bots follow it. Protect private areas with authentication.
Expecting llms.txt to replace it. A separate llms.txt file is optional, and Ahrefs found 97% of llms.txt files got zero requests. See whether llms.txt is worth it.

Setup steps for both files are in XML sitemaps and robots.txt for startups, and a robots.txt review is a standard line item in any SEO audit report.

Frequently asked questions

Where is the robots.txt file located?: It must sit at the root of the host, such as www.example.com/robots.txt. Crawlers don't look for it in subfolders, and each subdomain needs its own file, so blog.example.com and app.example.com are handled separately. Webflow, Framer and most hosts let you edit it in site settings, and Next.js can generate it from a robots file in the app directory.
Does robots.txt stop a page from appearing in Google?: Not reliably. Robots.txt stops Google from crawling a URL, but if other pages link to it, Google can still index the URL and show it without a description. To keep a page out of results, let Google crawl it and add a noindex meta tag or header, or remove the page and return a 404 or 410.
Do all bots follow robots.txt?: No. Major search engines and the main AI companies say their crawlers honor it, but the protocol is voluntary. Scrapers and malicious bots can ignore it entirely. If you need to stop a specific bot that ignores robots.txt, block it at the server or through your CDN's bot management, such as Cloudflare's controls.
How do I test my robots.txt file?: Open the file in a browser first and read it line by line. Then use the robots.txt report in Google Search Console, which shows the version Google fetched and any errors. Use the URL Inspection tool to check whether a specific page is blocked. Retest after every launch, platform change or edit.

Mehdi Hassan · CEO & Design Director

Copywriter, designer and developer, and founder of Designify Studio. Over five years, 60+ projects across 14+ industries, for startups that have raised $300M+.

Want this done for you? Website, $8,000.

A multi-page marketing site, written and designed by us, developed in Next.js and animated. $8,000, one-time, up to 6 pages. Book a free 30-minute call to see if it fits.

Schedule Call