AI crawler access
robots.txt rules for 23 AI bots, split into search indexes, live browsing agents and training crawlers. Firewall blocks of AI user agents. noindex and nosnippet directives.
Free tool
How do AI engines see your website?
Paste any URL. We fetch it exactly as an AI crawler does, then grade it on 44 signals: crawler access, JavaScript-free content, structured data, metadata and the content patterns that earn citations in ChatGPT, Perplexity, Claude and Google AI.
Try:
What it checks
Six areas, 44 signals.
Content without JavaScript
Most AI crawlers never run scripts. We measure the text that exists in the raw HTML, detect client-side rendered shells and check headings and landmarks.
Structured data & entity
Valid JSON-LD, an Organization or Person entity with sameAs links, a WebSite node and a specific page type so engines know who you are and what the page is.
Metadata & on-page
Title and description length, self-referencing canonical, language, Open Graph tags, mobile viewport and heading hierarchy.
GEO content signals
Answer-first openings, question-shaped headings, lists and tables, specific numbers, outbound citations, a named author, dates and depth. The patterns AI answers quote.
Technical
HTTPS, response time, redirect chains, sitemap discovery, image alt coverage and the emerging llms.txt convention.
Reading the report
How should you read the score?
The score is a weighted average of six category scores. Each of the 44 checks passes, warns or fails, and a warning earns half credit. A page scores 90 or above for grade A, 75 for B, 60 for C and 40 for D. Access and JavaScript-free content carry the most weight, 45 points between them, because nothing else matters if an engine cannot fetch and read the page.
- Start with the priority list. It ranks every failing or warning check by the points it costs, so the first three items are almost always the fastest way up.
- Check the crawler grid next. A blocked search crawler such as OAI-SearchBot or PerplexityBot removes the page from that engine's answers entirely. Blocked training crawlers are shown for information only and do not change the score.
- Read the "what an AI crawler reads" panel. If the text preview is empty or shows navigation only, the page is rendered by JavaScript and needs server-side rendering before anything else.
- Audit 3 to 5 pages, not one. The homepage, a service page and your best blog post usually show different problems. Structured data gaps tend to be site-wide; content gaps are page by page.
- Re-run after each deploy. The share link re-runs the audit live, so it doubles as a regression check.
Crawler behaviour is taken from the vendors' own documentation: OpenAI's crawler overview describes OAI-SearchBot, ChatGPT-User and GPTBot, and Google Search Central confirms that AI Overviews and AI Mode use the standard index with no special markup.
Questions
Good to know.
Is the AI SEO audit free?
Yes. Run it on as many pages as you like. The score, the six category scores, the priority fixes and the crawler grid need no sign-up. The check-by-check breakdown and the exports ask for an email address once, and stay unlocked in your browser after that.
What does the tool actually check?
It fetches the page with the same user agent OpenAI publishes for GPTBot, then scores 44 signals in six areas: AI crawler access, content available without JavaScript, structured data and entity clarity, metadata, GEO content patterns, and technical basics such as HTTPS, response time, sitemap and llms.txt.
Which AI crawlers does it evaluate?
Twenty-three, grouped by role: search indexes such as OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot; live browsing agents such as ChatGPT-User and Perplexity-User; and training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot. Blocking a search crawler lowers the score, blocking a training crawler does not.
Does it crawl my whole website?
No. It audits the single URL you enter, plus that site's robots.txt, sitemap and llms.txt. Run it on your homepage and on two or three of your most important pages for a representative picture.
Can it tell me whether ChatGPT cites my site?
Not on its own. The score reflects the on-page and access signals that make citation possible. Measuring where AI engines actually mention you needs prompt tracking, which is part of our full AI visibility audit.
Do you store the URLs or results?
Running an audit stores nothing: the result is returned to your browser only, and share links simply re-run it. If you unlock the full breakdown, we keep your email, the URL you audited and its score so we can follow up about the results. Nothing else.
Want the fixes done for you?
A full AI visibility audit covers your whole site, your competitors and where AI engines currently cite you.