Question 1

What is a robots.txt file?

Accepted Answer

robots.txt is a plain-text file placed at the root of your domain (e.g. https://example.com/robots.txt) that tells compliant web crawlers which pages or directories they are allowed or not allowed to access. It implements the Robots Exclusion Standard. It is a directive, not an enforcement mechanism — crawlers that follow the standard will respect it, but bad actors can ignore it. It does not substitute for access controls on sensitive pages.

Question 2

What is the difference between Allow and Disallow?

Accepted Answer

Disallow prevents a crawler from fetching the specified path. Allow (supported by Google, Bing, and others) overrides a broader Disallow for a more specific path — useful for blocking /private/ but explicitly allowing /private/public-doc.html. When both rules match a URL, the more specific path wins. An empty Disallow: (no path) means allow everything for that user-agent.

Question 3

What are GPTBot, ClaudeBot, and PerplexityBot?

Accepted Answer

These are the web crawlers run by OpenAI, Anthropic, and Perplexity respectively to collect training data and power their AI answer engines. GPTBot feeds ChatGPT and OpenAI model training; ClaudeBot is used by Anthropic for Claude; PerplexityBot powers Perplexity AI answers. If you want your content cited in AI-generated answers, allow these bots. If you want to prevent your content from being used in AI training datasets, block them with Disallow: /.

Question 4

Should I block or allow AI crawlers?

Accepted Answer

It depends on your goal. Block AI crawlers if you want to protect proprietary content from being used in training data without compensation or attribution. Allow AI crawlers if you want your brand, products, or content to appear in AI-generated answers (ChatGPT, Perplexity, Gemini) — increasingly where high-intent users find recommendations. For most brands, allowing AI crawlers on marketing and product content while blocking account/admin paths is the right default.

Question 5

Does robots.txt affect my Google rankings?

Accepted Answer

Yes, indirectly. If you block Googlebot from crawling a page, Google cannot index it and it will not appear in search results. A common mistake is disallowing /wp-admin/ but accidentally also blocking CSS/JS files — Google then cannot render the page properly, which hurts how it evaluates the page. Only block pages you genuinely do not want indexed (admin, staging, checkout, duplicate faceted URLs). Never block your CSS, JS, or image assets.

Question 6

What is Crawl-delay and should I use it?

Accepted Answer

Crawl-delay tells a crawler how many seconds to wait between consecutive requests to your server. Googlebot ignores it (use Google Search Console Crawl Rate settings instead); Bingbot and some other crawlers do respect it. Use it only if your server struggles with crawler load. A value of 1 to 10 seconds is typical. Setting it too high can slow down indexation of new or updated content.

Question 7

How do I add my sitemap to robots.txt?

Accepted Answer

Add a Sitemap: directive at the end of your robots.txt file: Sitemap: https://example.com/sitemap.xml. This tells crawlers where your sitemap lives regardless of which user-agent block they matched. You can include multiple Sitemap: lines. This is separate from — and does not replace — submitting your sitemap in Google Search Console.

Robots.txt Generator

Configure your robots.txt

Your robots.txt

robots.txt is your AI citation switch

Block to protect content

Allow to get cited

Mix allow and block

Frequently asked questions

Technical hygiene is step one. Revenue is the point.