--- title: "AI agent readiness checker" description: "Check how ready your site is for AI agents: robots.txt, llms.txt, markdown, Link headers, API catalog, OAuth, MCP and A2A cards, with a fix for each." url: "https://rookflare.com/tools/ai-readiness-checker/" --- # AI agent readiness checker See how well AI agents can find, read and use your site. We run 29 checks across discovery, markdown content, crawler rules, agent protocols, agent commerce and page signals, then give a score with a fix for each problem, quickest first. Use the form at https://rookflare.com/tools/ai-readiness-checker/ (needs JavaScript); the API endpoint is POST /api/public/v1/tools/ai-readiness with a JSON body { "url": "https://example.com" }. ## What this tool checks ### Discoverability robots.txt (RFC 9309), the XML sitemap, Link response headers with registered relations (RFC 8288) and DNS records for agent discovery (DNS-AID, an early IETF draft). ### Content for agents Markdown on request (Accept: text/markdown with Vary: Accept), a linked markdown alternate, /llms.txt in the llmstxt.org format with live links, and /llms-full.txt. ### Bot access Which AI crawlers your robots.txt allows or blocks, Content-Signal or Content-Usage preferences, and a Web Bot Auth key directory if you run a bot. ### Agent protocols API catalog (RFC 9727), OAuth and OpenID metadata (RFC 8414), protected resource metadata (RFC 9728), auth.md, MCP server cards, A2A agent cards, agent skills, ARD manifests, WebMCP forms and the Personal Agent Protocol. ### Agent commerce x402 and MPP payment challenges on the URL you enter, and the UCP and ACP discovery files. They only count when we find them. ### Page signals JSON-LD that parses, text in the raw HTML before any script runs, a canonical URL and the four required Open Graph tags. ## How to fix common problems ### Start with the quick fixes robots.txt, llms.txt, Content-Signal, the API catalog, canonical and Open Graph are single files or tags. The result lists them first, each with an example you can copy. ### No markdown version Publish a markdown copy of each page (for example /page/index.md), link it with rel="alternate" type="text/markdown" and answer Accept: text/markdown with it plus Vary: Accept. ### Little text without JavaScript Agents often read the raw HTML. Render the main content on the server (SSR or static generation) instead of only in the browser. ### Hand it to your coding agent Copy all fix instructions puts every failing check, the reason and an example into one prompt for Claude Code, Cursor or Copilot. Review what it changes before you merge. ## The standards behind each check Agent readiness is a mix of long-standing web standards and new proposals that are still changing. Each check below links its specification and says how settled it is, so you can decide what is worth doing now. ### Discoverability How agents find what you publish: the robots.txt file every crawler reads first, the sitemap that lists your pages, Link headers that point at machine-readable resources, and an early proposal for discovering agents through DNS. - robots.txt: Crawlers and agents read /robots.txt first to learn what they may fetch and where your sitemap is. A 5xx answer tells well-behaved crawlers to stay away from the whole site. [RFC 9309 (Robots Exclusion Protocol)](https://www.rfc-editor.org/rfc/rfc9309) (IETF standard) - XML sitemap: A sitemap lists every page you want found, so crawlers and agents do not have to discover them link by link. [Sitemaps protocol 0.9](https://www.sitemaps.org/protocol.html) (Community convention) - Link response headers: A Link header on the homepage points agents at machine-readable resources (API catalog, markdown version, llms.txt) without parsing any HTML. [RFC 8288 (Web Linking)](https://www.rfc-editor.org/rfc/rfc8288) (IETF standard) [IANA link relations](https://www.iana.org/assignments/link-relations/) (Published standard) - DNS records for agent discovery: DNS-AID proposes SVCB records under _agents. so agents can find your agent endpoints before loading any page. It is an early individual IETF draft. [DNS-AID (draft-mozleywilliams-dnsop-dnsaid)](https://datatracker.ietf.org/doc/draft-mozleywilliams-dnsop-dnsaid/) (IETF draft) ### Content for agents How agents read your pages. Markdown is smaller and cleaner than HTML, so agents ask for it when they can. llms.txt is a short markdown index of your site for language models. - Markdown on request (Accept: text/markdown): Agents that ask for text/markdown get clean text instead of HTML, which is smaller and easier to read correctly. [RFC 7763 (text/markdown)](https://www.rfc-editor.org/rfc/rfc7763) (IETF standard) [RFC 9110 (content negotiation, Vary)](https://www.rfc-editor.org/rfc/rfc9110) (IETF standard) - Markdown alternate link: A tells agents where the markdown version of this page lives. [llms.txt proposal](https://llmstxt.org/) (Community convention) - /llms.txt: llms.txt is a short markdown index of your site written for language models: what the site is and which pages matter. [llms.txt proposal](https://llmstxt.org/) (Community convention) - /llms-full.txt: Some tools read llms-full.txt, your docs in one markdown file. It is a community convention, not part of the llms.txt proposal. [llms.txt proposal](https://llmstxt.org/) (Community convention) ### Bot access Which AI crawlers may read your site and for what. robots.txt groups decide access per crawler; Content Signals and the IETF AI preferences drafts state how content may be used. Web Bot Auth lets bots prove who they are. - AI crawler rules: Naming AI crawlers in robots.txt makes your choice explicit per company and purpose (training, search, user requests). Allowing or blocking is your decision; we only report it. [RFC 9309 (Robots Exclusion Protocol)](https://www.rfc-editor.org/rfc/rfc9309) (IETF standard) - AI usage preferences: Content-Signal lines (a Cloudflare policy) or Content-Usage (an IETF draft) state whether content may be used for search, AI answers or AI training. Neither is enforced; they record your terms. [Content Signals Policy](https://contentsignals.org/) (Vendor proposal) [IETF AI Preferences vocabulary](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) (IETF draft) - Web Bot Auth key directory: Sites that run their own bots or agents publish signing keys so other sites can verify those requests. Only relevant if you operate a bot. [Web Bot Auth (draft-ietf-webbotauth-httpsig-protocol)](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/) (IETF draft) ### Agent protocols How agents find your APIs, sign users in and talk to your own agents or MCP servers. The OAuth documents and the API catalog are RFCs; the agent-specific formats are recent proposals. - API catalog: RFC 9727 gives every site one well-known address that lists its public APIs and their OpenAPI descriptions, so agents can find them without guessing. [RFC 9727 (api-catalog)](https://www.rfc-editor.org/rfc/rfc9727) (IETF standard) - OAuth / OpenID discovery: Agents that sign in on behalf of a user read your authorization server metadata to find the login and token endpoints. [RFC 8414 (OAuth server metadata)](https://www.rfc-editor.org/rfc/rfc8414) (IETF standard) [OpenID Connect Discovery 1.0](https://openid.net/specs/openid-connect-discovery-1_0.html) (Published standard) - OAuth protected resource metadata: RFC 9728 tells an agent which authorization server protects your API. MCP servers are required to publish it. [RFC 9728 (protected resource metadata)](https://www.rfc-editor.org/rfc/rfc9728) (IETF standard) - auth.md for agents: auth.md (a WorkOS proposal) is a markdown guide that tells agents how to authenticate, linked from your OAuth metadata. [auth.md](https://github.com/workos/auth.md) (Vendor proposal) - MCP server card: A server card describes your MCP server (name, version, endpoints) so clients can connect without trial and error. [MCP Server Cards (SEP-2127)](https://modelcontextprotocol.io/seps/2127-mcp-server-cards) (Vendor proposal) - A2A agent card: Agents that talk to other agents over the A2A protocol fetch /.well-known/agent-card.json to learn what your agent can do. [A2A protocol: Agent Card](https://a2a-protocol.org/latest/specification/) (Vendor proposal) - Agent skills index: A skills index lists SKILL.md files agents can download to learn how to use your product. [Agent Skills discovery (draft v0.2)](https://github.com/cloudflare/agent-skills-discovery-rfc) (Vendor proposal) - Agent resource catalog (ARD): An ARD manifest lists the agent-facing resources of a domain (MCP servers, agents, APIs) in one file. [Agentic Resource Discovery (ARD) v0.91](https://agenticresourcediscovery.org/spec) (Vendor proposal) - WebMCP tools in the page: WebMCP lets a page offer actions to browser agents. We can only see the declarative form attributes in the HTML; tools registered from JavaScript are invisible to this check. [WebMCP](https://github.com/webmachinelearning/webmcp) (W3C community draft) - Personal Agent Protocol: A draft protocol that lets personal agents discover how to sign in and act on your site for a user. [Personal Agent Protocol (v0.3 draft)](https://personalagentprotocol.org/) (Vendor proposal) ### Agent commerce How agents pay or check out. These protocols are new and mostly vendor-led; we only count them when your site publishes them. - x402 payments: x402 lets an agent pay for a resource over HTTP: the server answers 402 with what it accepts. There is no site-wide discovery file, so check the URL of a paid resource. [x402 specification](https://github.com/coinbase/x402/tree/main/specs) (Vendor proposal) - Machine payments (MPP): MPP is a draft HTTP authentication scheme for payments: a 402 with WWW-Authenticate: Payment, and an OpenAPI file that marks paid operations. [Payment HTTP authentication (MPP)](https://paymentauth.org/) (IETF draft) - Universal Commerce Protocol: A UCP profile tells shopping agents which commerce services and payment handlers your store supports. [Universal Commerce Protocol](https://ucp.dev/specification/overview/) (Vendor proposal) - Agentic Commerce Protocol: ACP defines a checkout API for agents. Its discovery file is still a proposal, so few stores publish it yet. [Agentic Commerce Protocol](https://github.com/agentic-commerce-protocol/agentic-commerce-protocol) (Vendor proposal) ### Page signals Signals inside the HTML that agents and search engines already rely on: structured data, text that does not need JavaScript, the canonical URL and Open Graph tags. - Structured data (JSON-LD): JSON-LD states facts about the page (who runs it, what is sold, prices) in a form agents and search engines read without guessing. [schema.org](https://schema.org/) (Community convention) [JSON-LD 1.1](https://www.w3.org/TR/json-ld11/) (Published standard) - Text without JavaScript: Many agents read the raw HTML and never run scripts. A page that renders its text only in the browser looks empty to them. - Canonical URL: The canonical tells agents and search engines which URL is the real one, so they do not cite a copy with tracking parameters. [RFC 6596 (canonical link)](https://www.rfc-editor.org/rfc/rfc6596) (IETF standard) - Open Graph tags: og:title, og:type, og:image and og:url give link previews and many agents a clean title, image and URL for the page. [Open Graph protocol](https://ogp.me/) (Vendor proposal) ## What Rookflare monitors today Continuous monitoring of these AI readiness checks is not part of Rookflare today. What Rookflare monitors today: desktop and mobile screenshots of the pages you choose compared with your approved baseline, HTTP status, redirects, broken images, JavaScript errors, critical elements, a noindex tag or missing title, title and canonical changes, and on Growth and Agency the uptime of your key URLs. [Start free trial](https://app.rookflare.com/register/). ## Frequently asked questions ### How is the score calculated? Each check has a weight that depends on the site type you pick. A pass earns the full weight, a warning half and a fail nothing. Checks marked not applicable are left out, so a blog is not marked down for having no payment API. The level reads 85 and up as Agent-ready, 60 as Agent-friendly, 35 as Partly ready. ### Which site type should I pick? Content site for blogs, docs, shops and marketing sites: reading and discovery weigh most, and API or agent protocols only count when you publish them. API or app for products with an API, sign-in or an MCP server: the API catalog, OAuth metadata and MCP card count even when missing. All checks counts everything. ### Are all of these official standards? No. robots.txt, Link headers, the API catalog and the OAuth documents are IETF RFCs. DNS-AID, Web Bot Auth, AI usage preferences and MPP are IETF drafts, WebMCP is a W3C community draft, and llms.txt, Content Signals, MCP server cards, A2A, agent skills, ARD, x402, UCP and ACP are vendor or community proposals. Each check in the result names its status and links the specification. Drafts change, so we only count most of them when you publish them. ### Why is x402 not applicable on my homepage? x402 has no site-wide discovery file. A paid resource answers 402 with its payment requirements, so enter the URL of a paid endpoint to test it. ### Does a high score mean AI can train on my content? No. The score is about agents finding and reading your pages. We report which AI crawlers your robots.txt allows or blocks without judging the choice, and training is a separate preference (Content-Signal: ai-train=no). ### How do you look up DNS records? Through a public DNS-over-HTTPS resolver, which only answers public names. We query _agents and _index._agents under your domain and nothing else. ### Can I share a result? Yes. After a check the page address includes your URL and options. Anyone who opens it runs the same check again; within 10 minutes they get the saved result. We keep no history. ### Is this tool free? Yes. You do not need an account. Each IP address can run 10 checks a minute and 100 a day, and a website can be checked 10 times in 10 minutes across all visitors so the tool is never used to flood a site. ### Do you store the URL or the results? The result is kept for 10 minutes so a repeated check is instant, then it is deleted. We do not log the URLs you check or the content of the page. ## More free tools - [Robots.txt and sitemap validator](https://rookflare.com/tools/robots-txt-validator/): Crawler rules, AI bots, sitemap files and URL count. - [Meta tag and Open Graph checker](https://rookflare.com/tools/meta-tag-checker/): Title, description, canonical and social card preview.