Input: one authorized public origin and a small list of product, pricing, documentation, and policy URLs. Output: a dated access report for those URLs. This procedure does not authorize private access, scanning unrelated systems, bypassing challenges, or changing a site's crawl policy.
Define the scope
Record the origin, task, URLs, request budget, intended consumer, and collection time. Use public pages without credentials. Identify which requests are ordinary retrieval checks and which are owner-approved user-agent rule simulations. Do not label simulations as requests from the real provider.
Collect the response and the content
The following commands are templates for a site you are allowed to inspect. The example domain is a placeholder. They make bounded GET requests and do not follow redirects automatically.
curl --proto '=https' --max-time 15 --max-filesize 2097152 \
--dump-header robots.headers --output robots.txt \
'https://example.com/robots.txt'
curl --proto '=https' --max-time 15 --max-filesize 2097152 \
--dump-header product.headers --output product.html \
'https://example.com/product'Inspect redirects before fetching their targets. Reapply the scope check at every hop. If implementing this as a service, validate resolved addresses and redirect destinations, exclude private and metadata networks, constrain decompressed size, and use an egress boundary. A URL string beginning with HTTPS is not sufficient SSRF protection.
Separate the access layers
| Layer | Inspect | Interpretation |
|---|---|---|
| Transport | Status, TLS failure, redirect, timeout | What this request could reach |
| Crawl policy | Applicable robots group and path rules | Published rules for that crawler, not authentication |
| Indexing | Meta robots and X-Robots-Tag | Indexing directives, subject to the consumer's behavior |
| Delivery | Challenge, login, consent wall, empty shell | Whether this response contains useful content |
| Meaning | Product, constraints, price unit, next action | Whether the task can be answered from the source |
Do not infer a universal indexing state from a robots file. A robots disallow can prevent a crawler from seeing a page's noindex directive. Confirm intended behavior using the relevant provider's documentation and webmaster tools where available.
Verify identity separately
Use the provider's published verification mechanism for observed traffic. A user-agent string is a claim, not authentication. OpenAI, Anthropic, and Perplexity publish different crawler roles and controls; the platform guide links the authoritative references. Do not hardcode an IP list copied from an old article.
Scope any proposed firewall change to the permitted public route and verified source. Preserve protections around accounts, payments, uploads, and private APIs. Record which control caused the failure before suggesting a repair.
Report what happened
{
"example": true,
"url": "https://example.com/product",
"method": "GET",
"collection_mode": "ordinary_public_fetch",
"observed_at": null,
"http_status": null,
"useful_content_present": null,
"provider_identity_verified": false,
"indexing_status": "unknown",
"proposed_check": "Confirm the product facts exist in the response body."
}Null means uncollected, not a failing response or a zero value. After a repair, repeat the same scoped request and retain both observations. A working fetch proves access from that test environment. Provider crawling, indexing, citation, and customer conversion require additional evidence. Link the finding to an exact response or excerpt rather than an unsupported readiness percentage.