Skip to content
Start free trial

Robots.txt and sitemap validator

Read a site’s robots.txt line by line and check the sitemaps it lists.

A public address such as example.com or https://example.com/pricing/.

What you get

  • Every rule, annotated in plain words
  • Which AI crawlers may fetch the homepage
  • Sitemap files and how many URLs they list
  • A status check of sampled sitemap URLs

Mistakes

Common robots.txt mistakes

Each one is a single line that is easy to miss in a review.

Staging rules shipped to production

Disallow: / for every crawler stops search engines from crawling the whole site.

Before

  1. User-agent: *
  2. Disallow: /

After

  1. User-agent: *
  2. Disallow: /checkout/
  3. Disallow: /cart/

Hiding a page with robots.txt

A blocked page can still be indexed from links, without its content. To keep it out of results, let crawlers fetch it and add a noindex robots meta tag.

Before

  1. User-agent: *
  2. Disallow: /thank-you/

After

  1. User-agent: *
  2. Allow: /
  3. # /thank-you/ sends <meta name="robots" content="noindex">

A bot group that forgets the shared rules

A crawler follows only the group that names it, so the rules under User-agent: * no longer apply to it. Repeat the shared rules in its group.

Before

  1. User-agent: *
  2. Disallow: /checkout/
  4. User-agent: GPTBot
  5. Disallow: /drafts/

After

  1. User-agent: *
  2. Disallow: /checkout/
  4. User-agent: GPTBot
  5. Disallow: /drafts/
  6. Disallow: /checkout/

A relative sitemap address

The Sitemap line needs a full URL, including https://.

Before

  1. Sitemap: /sitemap.xml

After

  1. Sitemap: https://northwind-shop.example/sitemap.xml

AI crawlers

Training, search and user agents are different bots

AI companies run separate agents for model training, for their search index and for fetching a page when someone asks about it. Each can be allowed or blocked on its own.

Kind User agents What blocking does
Training GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended Asks that your pages are not used to train models. Google-Extended and Applebot-Extended are control tokens, not crawlers: they cover pages Google and Apple already crawl for search.
Search OAI-SearchBot, Claude-SearchBot, PerplexityBot Can keep your pages out of AI search answers and their source links.
User request ChatGPT-User, Claude-User, Perplexity-User These fetch one page when a person asks about it. OpenAI and Perplexity say these user-started fetches may not follow robots.txt.

What this tool checks

  • robots.txt Found or not, file size against Google’s 500 KiB limit, user-agent groups and their rules, lines we could not read and Content-Signal lines.
  • AI crawlers For GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others: which group applies and whether the homepage is allowed.
  • Sitemaps Sitemap files from robots.txt or /sitemap.xml, whether they parse, how many URLs they list, and a status check of 10 of those URLs.

How to fix common problems

Disallow: / for everyone
Often left over from staging. It stops search engines from crawling the site. Remove it or limit it to the paths that should stay private.
Sitemap not declared
Add a line such as Sitemap: https://example.com/sitemap.xml so every crawler finds it, not only the ones you submitted it to.
Sitemap lists broken or redirected URLs
A sitemap should only list final, indexable URLs that return 200. Regenerate it after removing or moving pages.

FAQ

Questions

Something missing here? Send us a message

How do I block AI training but stay in AI search?

Add a group for training crawlers such as GPTBot, ClaudeBot and Google-Extended with Disallow: /, and leave search agents such as OAI-SearchBot and Claude-SearchBot allowed. A Content-Signal line states the same preference in one place.

How many URLs do you read?

Up to 5 sitemap files and 5,000 URLs. A larger sitemap is marked as truncated; the files we read are still checked.

Is this tool free?

Yes. You do not need an account. Each IP address can run 10 checks a minute and 100 a day, and a website can be checked 10 times in 10 minutes across all visitors so the tool is never used to flood a site.

Do you store the URL or the results?

The result is kept for 10 minutes so a repeated check is instant, then it is deleted. We do not log the URLs you check or the content of the page.

More free tools

Start monitoring your website

Add your address, pick the pages and approve the first scan as your baseline.

Start monitoring Talk to us

Plans start at €19 per month. Prices are provisional.