--- title: "Robots.txt and sitemap validator" description: "Validate robots.txt and XML sitemaps: crawler groups, AI bot rules, sitemap URLs, URL count and a sample of sitemap pages that return errors." url: "https://rookflare.com/tools/robots-txt-validator/" --- # Robots.txt and sitemap validator Check robots.txt and the XML sitemaps it lists: crawler groups, rules for AI crawlers, sitemap files, the number of URLs and a sample of sitemap pages that fail. Use the form at https://rookflare.com/tools/robots-txt-validator/ (needs JavaScript); the API endpoint is POST /api/public/v1/tools/robots with a JSON body { "url": "https://example.com" }. ## What you get - Every rule, annotated in plain words - Which AI crawlers may fetch the homepage - Sitemap files and how many URLs they list - A status check of sampled sitemap URLs ## Common robots.txt mistakes Each one is a single line that is easy to miss in a review. ### Staging rules shipped to production Disallow: / for every crawler stops search engines from crawling the whole site. Before: ``` User-agent: * Disallow: / ``` After: ``` User-agent: * Disallow: /checkout/ Disallow: /cart/ ``` ### Hiding a page with robots.txt A blocked page can still be indexed from links, without its content. To keep it out of results, let crawlers fetch it and add a noindex robots meta tag. Before: ``` User-agent: * Disallow: /thank-you/ ``` After: ``` User-agent: * Allow: / # /thank-you/ sends ``` ### A bot group that forgets the shared rules A crawler follows only the group that names it, so the rules under User-agent: * no longer apply to it. Repeat the shared rules in its group. Before: ``` User-agent: * Disallow: /checkout/ User-agent: GPTBot Disallow: /drafts/ ``` After: ``` User-agent: * Disallow: /checkout/ User-agent: GPTBot Disallow: /drafts/ Disallow: /checkout/ ``` ### A relative sitemap address The Sitemap line needs a full URL, including https://. Before: ``` Sitemap: /sitemap.xml ``` After: ``` Sitemap: https://northwind-shop.example/sitemap.xml ``` ## Training, search and user agents are different bots AI companies run separate agents for model training, for their search index and for fetching a page when someone asks about it. Each can be allowed or blocked on its own. | Kind | User agents | What blocking does | | --- | --- | --- | | Training | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended | Asks that your pages are not used to train models. Google-Extended and Applebot-Extended are control tokens, not crawlers: they cover pages Google and Apple already crawl for search. | | Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Can keep your pages out of AI search answers and their source links. | | User request | ChatGPT-User, Claude-User, Perplexity-User | These fetch one page when a person asks about it. OpenAI and Perplexity say these user-started fetches may not follow robots.txt. | ## What this tool checks ### robots.txt Found or not, file size against Google’s 500 KiB limit, user-agent groups and their rules, lines we could not read and Content-Signal lines. ### AI crawlers For GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others: which group applies and whether the homepage is allowed. ### Sitemaps Sitemap files from robots.txt or /sitemap.xml, whether they parse, how many URLs they list, and a status check of 10 of those URLs. ## How to fix common problems ### Disallow: / for everyone Often left over from staging. It stops search engines from crawling the site. Remove it or limit it to the paths that should stay private. ### Sitemap not declared Add a line such as Sitemap: https://example.com/sitemap.xml so every crawler finds it, not only the ones you submitted it to. ### Sitemap lists broken or redirected URLs A sitemap should only list final, indexable URLs that return 200. Regenerate it after removing or moving pages. ## Monitor this automatically Rookflare reads your sitemap when you add a website so you can pick the pages to watch, then checks each of them for server errors and noindex on every scan. [Start free trial](https://app.rookflare.com/register/). ## Frequently asked questions ### How do I block AI training but stay in AI search? Add a group for training crawlers such as GPTBot, ClaudeBot and Google-Extended with Disallow: /, and leave search agents such as OAI-SearchBot and Claude-SearchBot allowed. A Content-Signal line states the same preference in one place. ### How many URLs do you read? Up to 5 sitemap files and 5,000 URLs. A larger sitemap is marked as truncated; the files we read are still checked. ### Is this tool free? Yes. You do not need an account. Each IP address can run 10 checks a minute and 100 a day, and a website can be checked 10 times in 10 minutes across all visitors so the tool is never used to flood a site. ### Do you store the URL or the results? The result is kept for 10 minutes so a repeated check is instant, then it is deleted. We do not log the URLs you check or the content of the page. ## More free tools - [AI agent readiness checker](https://rookflare.com/tools/ai-readiness-checker/): llms.txt, markdown, crawler rules and agent protocols. - [Meta tag and Open Graph checker](https://rookflare.com/tools/meta-tag-checker/): Title, description, canonical and social card preview.