Can an agent actually read your website?

Before optimizing what an agent learns, check whether it can retrieve the information at all. A page that works in your logged-in browser can still return a challenge, empty shell, or access error to another client. Test the public path from outside your session.

Check the response, then the content

Start with the important pages: product, pricing, documentation, policies, and the next action. Confirm that each returns the intended status and useful text without requiring a personal account. Do not mistake a 200 response containing a login page or bot challenge for accessible product information.

Google can render JavaScript, but not every crawler or retrieval client does. Server-rendered text and real links reduce the work required to access a page. A separate Markdown version can help some tools; it should carry the same facts as the HTML. Google’s JavaScript SEO guidance.

Understand the controls

Robots rules express crawler preferences; they are not access control. Authentication and authorization must protect private content. The Robots Exclusion Protocol defines how participating crawlers interpret rules, but does not make private data safe to publish. RFC 9309.

Search crawling and model training are also different uses. OpenAI documents separate controls for OAI-SearchBot and GPTBot. ChatGPT-User represents certain user-initiated actions and is not the automatic search crawler. Choose policies deliberately instead of assuming one “AI bot” switch controls every use. OpenAI crawler reference.

A practical inspection sequence

• Fetch each priority URL without your logged-in cookies. Inspect both the response code and body.
• Confirm the main answer, product facts, and links exist in the initial HTML or a documented text alternative.
• Review robots rules, page-level indexing directives, and CDN or firewall challenges together.
• Check that canonical URLs identify the intended public page and do not point to a staging domain.
• Link important pages from ordinary navigation or contextual links; do not rely on a search box to expose them.
• Include canonical public URLs in a sitemap and update modification dates only when the content changes.

A sitemap helps a search engine discover URLs; it does not guarantee that they will be indexed. Avoid stuffing it with query results, duplicates, or pages that redirect. Google’s sitemap overview.

Diagnose the layer that failed

A DNS or TLS problem prevents a connection. A firewall challenge can prevent retrieval. A robots rule can exclude a cooperative crawler. An empty JavaScript shell can prevent extraction. Contradictory canonical signals can confuse indexing. These failures need different fixes; adding more copy will not resolve all of them.

Keep the test honest

Do not weaken authentication on customer or administrative pages to improve crawlability. Publish a separate public explanation of the offering. A successful anonymous fetch establishes access for that request, not adoption by an AI platform. Verify the next layer independently: whether a relevant system encounters and uses the information.

Diagnose a failure without guessing

A 200 response can still be an empty application shell or a challenge page. Inspect the body for the information the task needs. Keep crawl policy, indexing directives, firewall behavior, and authentication as separate checks. A request from your own environment cannot prove that a named provider has indexed the page.

A crawler user-agent string is not verified identity. Do not grant broad access to private routes because a request contains a familiar name. Use the relevant provider's verification guidance and scope any exception to intended public content. The platform comparison explains differences between search, training, and user-requested retrieval; the audit recipe provides bounded requests and an evidence record.