core.robots_bots
robots.txt must permit GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. A site that blocks these crawlers cannot be read or cited by AI models.
- Blocked Bots
- []
- Robots Txt Present
- yes
18 of 22 checks passing. Items marked “Reporting only” are shown for your information and do not change your grade.
Can AI assistants reach and read your site at all?
What it does: Checks that GPTBot, ClaudeBot and other AI crawlers are not blocked.
Why it matters: Blocked crawlers cannot learn about your business.
What it does: Checks the page source can be read without scripts.
Why it matters: Messy or script-only pages are skipped or misread.
What it does: Checks for a sitemap with canonical page addresses.
Why it matters: Helps crawlers find every page you want known.
What it does: Looks for a plain-text summary file written for AI models.
Why it matters: Gives assistants a clean overview to quote instead of guessing.
What it does: Checks the site can return a lightweight Markdown version of pages.
Why it matters: Cheaper and cleaner for AI to read.
Your result: Serving clean markdown on Accept: text/markdown content negotiation makes a site's content directly consumable by AI crawlers.
What it does: Looks for the longer companion to llms.txt.
Why it matters: Lets AI read your key content in one request.
What it does: Checks the text is present before any JavaScript runs.
Why it matters: Most AI crawlers do not run JavaScript, so script-only content is invisible.
Is your content structured so AI can quote it correctly?
What it does: Checks for machine-readable details about your business.
Why it matters: Structured data is the clearest way to tell AI who you are.
What it does: Checks your business name and details are the same everywhere.
Why it matters: Inconsistent names split your identity across sources.
What it does: Measures how many verifiable statements the page makes.
Why it matters: Pages with substance are more quotable.
What it does: Checks claims come with specifics.
Why it matters: Concrete facts get cited, vague claims do not.
What it does: Checks headings split content into self-contained blocks.
Why it matters: Blocks can be lifted out and quoted cleanly.
What it does: Looks for dates and update markers.
Why it matters: Fresh content is preferred over stale.
Your result: The page should carry fresh, dateable temporal markers (dates, <time>, datePublished/dateModified, last-modified) so answer engines know the content is current.
What it does: Looks for pages nothing links to.
Why it matters: Pages with no links in are rarely found or cited.
Your result: Sitemap pages with no internal link pointing to them are invisible to AI crawlers that navigate links instead of reading the sitemap.
What it does: Compares the structured data to the visible text.
Why it matters: Mismatches reduce trust in your data.
What it does: Looks for plain statements of what you do and where.
Why it matters: AI quotes direct statements.
What it does: Checks the answer comes near the top of the page.
Why it matters: AI favors direct answers up front.
Your result: Direct answers should sit near the top of the page, where answer engines actually anchor their responses.
What it does: Looks for references, numbers and sources.
Why it matters: Evidence makes claims believable to AI.
Your result: Specific claims should appear in paragraphs that carry an outbound link, so a reader or AI can verify the claim — citation-readiness starts with claims near sources.
How the page looks and works for people, phones and shared links.
What it does: Checks the page declares a mobile viewport.
Why it matters: Most visitors arrive on a phone; without it the page renders tiny.
What it does: Checks the tags that control how your link looks when shared.
Why it matters: Links with a picture and title get clicked; bare URLs do not.
What it does: Checks images carry alt text.
Why it matters: Helps screen readers and tells AI what images show.
What it does: Finds inputs with no label.
Why it matters: People using screen readers or voice cannot tell what to enter.
What it does: Checks for one main heading and a logical outline.
Why it matters: A clear outline helps people and AI skim the page.
Signals that show who you are and that you can be believed.
What it does: Looks for links to privacy and terms pages.
Why it matters: Visitors, app stores and ad platforms expect them.
Your result: The homepage should link to a privacy policy and terms; app stores, payment providers and trust signals expect them.
robots.txt must permit GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. A site that blocks these crawlers cannot be read or cited by AI models.
Content must be served as clean HTML in the initial response. If the page needs a headless browser to render, most AI crawlers never see the content.
schema.org structured data must be present and valid. It feeds grounded retrieval and lets models resolve what kind of entity you are.
Your name, description, and address must be identical across every owned property. Inconsistencies make AI models hedge or merge you with another business.
The homepage should contain specific, extractable facts (numbers, dates, quantities, credentials) rather than bare adjectives — models extract claims.
The homepage needs a self-consistent rel=canonical link and a sitemap.xml that lists the site's own URLs.
An llms.txt file should exist at /llms.txt and be well-formed (a title, sections, and real URLs) so AI crawlers can index the site's intent.
The page should answer likely queries directly (FAQ/HowTo structured data or question-shaped headings) so answer engines can ground a response in your content.
RAG retrieval performs best with self-contained 200–400-word sections under a coherent heading hierarchy (one h1, ≥2 h2/h3, no skipped levels).
The page should carry fresh, dateable temporal markers (dates, <time>, datePublished/dateModified, last-modified) so answer engines know the content is current.
Sitemap pages with no internal link pointing to them are invisible to AI crawlers that navigate links instead of reading the sitemap.
Serving clean markdown on Accept: text/markdown content negotiation makes a site's content directly consumable by AI crawlers.
An llms-full.txt file should exist at /llms-full.txt with the full site copy as markdown so AI crawlers can ground answers in every page.
Structured data must mirror the content a visitor actually reads. Schema that describes nothing on the page is misleading to grounded retrieval.
Entity schema should carry connectivity signals (org type, sameAs, author, org URL), not just a bare name — an entity graph is what lets AI resolve you unambiguously.
Direct answers should sit near the top of the page, where answer engines actually anchor their responses.
Specific claims should appear in paragraphs that carry an outbound link, so a reader or AI can verify the claim — citation-readiness starts with claims near sources.
Key content (title, headings, real body text) must be present in the raw HTML as served. AI crawlers that do not run JavaScript cannot read a content-less shell.
Pages need a mobile viewport tag (width=device-width) and must not disable pinch-zoom, or phones show a shrunken desktop page.
A page title, meta description and Open Graph title, description and image control how the page looks when shared and summarized.
Every image needs an alt attribute (empty alt is fine for decoration) so screen readers and AI crawlers can understand it.
Every visible form field needs a label (label-for, a wrapping label, or aria-label) so it can be used with assistive tech and voice.
A page needs exactly one h1 and headings that do not skip levels, so the outline is clear to readers and crawlers.
The homepage should link to a privacy policy and terms; app stores, payment providers and trust signals expect them.
A separate read of the content only, against the 7-dimension CITABLE rubric: 85 / 100.
| Dimension | Status | Coverage | Score |
|---|---|---|---|
| Clear entity/structure | Pass | 2 / 2 checks | 100 / 100 |
| Intent architecture | Pass | 3 / 3 checks | 100 / 100 |
| Third-party validation | Fail | 1 / 2 checks | 50 / 100 |
| Answer grounding | Fail | 2 / 3 checks | 86 / 100 |
| Block-structured for RAG | Pass | 1 / 1 checks | 100 / 100 |
| Latest & consistent | Not run | no data | - |
| Entity graph + JSON-LD | Pass | 4 / 4 checks | 100 / 100 |