Robots.txt Generator

How robots.txt works

robots.txt lives at the root of your domain (https://example.com/robots.txt) and tells crawlers which paths they may fetch. Each block starts with one or more User-agent lines followed by Allow/Disallow rules; Disallow: with an empty value permits everything, Disallow: / blocks the whole site. A Sitemap: line may appear anywhere. Note that robots.txt controls crawling, not indexing — a blocked page can still appear in results if other sites link to it; use a noindex meta tag to keep pages out of search.

Should you block AI crawlers?

It's a trade-off. Blocking GPTBot, ClaudeBot or Google-Extended keeps your content out of future model training runs, but may also remove you from AI search answers and assistants that cite sources — an increasingly large traffic channel. Blocking Google-Extended does not affect your normal Google Search ranking; it only opts out of Gemini training. If you'd rather guide AI systems to your best content than block them, generate an llms.txt file instead — the two files work well together.

Frequently asked questions

Where do I put the file?

At the domain root — it must be reachable at /robots.txt exactly. Subdirectory copies are ignored.

Do all bots obey robots.txt?

Reputable crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot) do. Malicious scrapers ignore it — robots.txt is a signpost, not a lock.

Is Disallow case-sensitive?

Yes — /Admin/ and /admin/ are different paths to a crawler.