# What llms.txt and Markdown can—and cannot—do > Use lightweight text as a helpful documentation interface, without treating it as an AI ranking switch. Canonical: https://byagents.dev/humans/llms-txt-and-markdown Published: 2026-09-13 Reviewed: 2026-09-13 Publisher: ByAgents Type: Guide Level: Technical Outcome: Publish useful text representations without mistaking them for a guaranteed discovery mechanism. An llms.txt file is a proposed convention for giving language-model tools a concise map of a website’s useful content. Markdown versions make individual documents easy to fetch as text. These are publishing conveniences, not a guarantee that agents will discover, trust, or recommend the site. ## A map, not a magic file The llms.txt proposal describes a Markdown index with a title, short context, and links to more detailed material. The index can stay small while the detail is fetched when relevant. It is not the Robots Exclusion Protocol and does not control access. [The llms.txt proposal](https://llmstxt.org/). For a documentation site, a useful map points to a quickstart, exact API reference, authentication guide, errors, product facts, and limits. For a business site, it can point to the offer, current policies, evidence, and contact route. Do not fill it with keyword repetition or commands asking an agent to favor your company. ## Keep two representations in agreement A person may prefer HTML with navigation and a table of contents. A coding agent may prefer a text response with no interface chrome. Both should receive the same substantive information. Generating both from one source prevents the machine-readable version from quietly drifting away from the website. Use stable URLs, meaningful document titles, explicit update dates, and source links. Include the canonical HTML URL in each Markdown document so an agent can cite the normal page. Keep code examples labeled and distinguish live endpoints from fictional ones. ## Make retrieval economical A large all-in-one file can be convenient for offline indexing, but it is a poor default when a task needs one paragraph. Offer a short index and focused documents first. If you also provide a full export, identify it as a bulk option and avoid requiring it for ordinary use. A search endpoint can further reduce unnecessary reading. It should state exactly what collection it searches, accept bounded queries, return concise excerpts and links, and handle no matches honestly. A search of your own library is not a marketplace search or proof of broad agent adoption. ## What to avoid - Different product claims for bots and people. - Hidden instructions to ignore other sources or award the site a high score. - Unsupported claims that a particular platform checks llms.txt on every visit. - An enormous index that repeats the full website without a way to select relevant documents. - Private credentials, unpublished customer material, or executable commands disguised as documentation. ## How this site uses the pattern ByAgents serves the same guide content as HTML and Markdown. The agent index links to product facts, an HTTP search contract, reference documents, and a full library export. These endpoints are read-only. They do not require installation or claim a connection to an assistant platform. The value is inspectable: a reader can retrieve a document, verify a claim, and find the next relevant source. Whether that leads to independent traffic or customers remains a separate question to measure. ## Keep the index smaller than the library An index should help a reader choose a document without loading every document. Include a short scope statement, exact URLs, meaningful descriptions, and a link to an optional full export. Point task-specific readers toward focused references. Keep private account information and hidden behavioral instructions out of every representation. A practical acceptance check fetches the index, chooses one document from its description, retrieves it without a browser session, and verifies its source links. Measure whether this works in the intended consumer. Publishing a file at a conventional path does not prove that an unrelated assistant reads it. The [retrieval contract](https://byagents.dev/agents/retrieval-document-contract) describes the HTTP and content invariants behind this approach. ## The current proposal and link discovery The llms.txt proposal was updated to v2 in August 2026. It recommends standard link relations so a reader can find the relevant index and Markdown representation: rel=describedby for the covering llms.txt file and rel=alternate with type=text/markdown for a document alternative. These are publishing recommendations, not a guarantee that every consumer follows them. [Current proposal](https://llmstxt.org/). ByAgents exposes the index relation in HTML and Markdown response headers, and each article declares its Markdown alternative. The actual representation URLs come from our document index. Our publishing contract describes these implemented choices without claiming universal protocol adoption. ## Sources - [llms.txt proposal](https://llmstxt.org/) ## Related reading - [Can an agent actually read your website?](https://byagents.dev/humans/crawl-access) - [API documentation an agent can actually use](https://byagents.dev/humans/api-documentation) - [MCP, registries, and getting your tools discovered](https://byagents.dev/humans/mcp-and-tool-discovery) - [Pattern: publish documents for reliable retrieval](https://byagents.dev/agents/retrieval-document-contract)