Agent-Readable Documentation and Context Packs

Most product and developer documentation is still designed for browser navigation. It has sidebars, search pages, redirects, version pickers, marketing pages, changelogs, API references, and support articles all mixed into one human-facing surface.

That is not the same thing as agent-readable context. An agent can fetch pages, but that does not mean it knows which pages are canonical, which ones are stale, which ones are safe to use, which ones are public, or which ones should be ignored. A sitemap is too broad. A raw crawl is too noisy. A copied documentation dump becomes stale quickly.

This issue builds a markdown-first documentation control package: a public llms.txt index, an optional expanded context pack, a source map, a freshness policy, an exclusions policy, and evaluation questions that test whether the pack actually supports grounded answers.

The Artifact

You are building a documentation surface that agents can inspect without treating the entire website as trusted context:

• A short llms.txt file that points to canonical documentation sources
• An optional llms-full.txt file for expanded, governed context loading
• A context pack contract that defines audience, scope, source classes, and failure modes
• A source map that records ownership, sensitivity, freshness, and inclusion mode
• A freshness policy that decides when a source is too stale to use
• An exclusions policy that blocks secrets, customer data, private links, and unstable material
• Evaluation questions that test whether answers are grounded in included sources

This is a markdown-first issue because the artifact itself is the product. Before a team needs a custom crawler, embedding job, or retrieval pipeline, it needs to decide which documentation is allowed to become agent context at all.

Not a Magic Standard

llms.txt should be framed carefully. The current llms.txt proposal describes a Markdown file at /llms.txt that helps language models use a website at inference time. It proposes a simple format: one required project heading, then optional summary, guidance, and sectioned lists of links to useful files.

That is useful. It is also not access control, not authentication, not a replacement for robots.txt, not a sitemap, and not a guarantee that every model provider, search product, crawler, coding agent, or browser assistant will read the file.

The right production framing is simple: llms.txt is a curated entry point. The system around it still owns source selection, privacy boundaries, freshness, evaluation, and runtime retrieval behavior.

Why This Matters Now

The pattern is no longer theoretical. Developer documentation sites now expose agent-readable entry points and Markdown views in production. The Model Context Protocol documentation publishes a llms.txt index. Cloudflare's developer documentation exposes a documentation-level llms.txt file. Stripe's docs expose llms.txt and also document machine-readable resources such as Markdown output for docs pages on its Agents and AI page. Vercel and Anthropic also expose public /llms.txt endpoints.

The important point is not that every implementation is identical. They are not. The point is that agent-readable documentation is becoming a real product surface. Once that happens, the engineering question changes from "can an agent crawl our docs?" to "which documentation contract are we exposing to agents, and how do we know it is correct?"

System Structure

The repository keeps the system intentionally small. The public index is separate from the expanded pack. The source map is separate from both. Freshness and exclusions are explicit policy files. Evaluation questions sit beside the pack instead of living in a separate testing idea that nobody runs.

That structure is deliberately boring. Each file has one job. The public index points. The full pack expands. The context contract defines boundaries. The source map records ownership. The policies decide what is safe and current. The eval questions check whether the pack works.

The Public Index

The llms.txt file should be short enough to review. If it becomes a second documentation site, it loses its value. The purpose is to point agents toward the smallest useful set of canonical sources.

The repo starts with this shape:

# Agent-Readable Documentation Context Pack

> Template for a curated agent-readable documentation index. Replace all
> placeholder links with real canonical documentation URLs before publishing.

This file follows the shape of the llms.txt proposal: one project heading, a
short summary, optional guidance, and sectioned lists of documentation links.
It is a template, not a live product index.

Use this file to point agents at the smallest set of sources needed to
understand the documentation surface. Do not use it as a full documentation
dump, an access-control mechanism, or a crawler policy.

## Core Context

- [Context pack contract](https://docs.example.com/CONTEXT_PACK.md): Defines
audience, scope, assembly order, source trust, and governance for the pack.
- [Source map](https://docs.example.com/docs/source-map.md): Records source
ownership, freshness rules, sensitivity, and canonical locations.
- [Freshness policy](https://docs.example.com/docs/freshness-policy.md):
Defines stale-source handling and update requirements.
- [Exclusions](https://docs.example.com/docs/exclusions.md): Defines content
that must not be included in public or internal agent context.

## Evaluation

- [Evaluation questions](https://docs.example.com/docs/eval-questions.md):
Evidence-based questions used to test whether the context pack is complete,
current, and safe enough for agent use.

## Optional

- [Expanded context pack](https://docs.example.com/llms-full.txt): Optional
concatenated context for offline inspection, large-context workflows, or
controlled internal loading.

The example uses docs.example.com because it is a template. In a real documentation site, those links should become canonical HTTPS URLs or stable internal URLs behind the right access boundary.

The Full Pack Is Optional

The base llms.txt proposal does not require a file named llms-full.txt. The proposal itself says it does not prescribe one processing method. Some projects generate expanded context files from their index. That is a useful pattern, but it should be treated as a governed artifact, not as a required part of the format.

The repo makes that boundary explicit:

# Agent-Readable Documentation Context Pack - Expanded Template

This file is an optional expanded context pack for workflows that need more
than the public llms.txt index.

The base llms.txt proposal does not require this exact file name. This
repository uses llms-full.txt as a practical convention for a larger,
governed context artifact.

Do not publish this file until the sources, exclusions, ownership, and
freshness rules have been reviewed.

That caveat matters. A public index can be low-risk when it mostly points to already public documentation. A full pack can carry much more content. It needs stricter review because stale material, private links, accidental secrets, and generated summaries become easier to miss when the file grows.

Source Map Before Content

A context pack without a source map is just a curated scrape. It may work for a demo, but it has no ownership model. Nobody knows who approves a source, when it expires, whether it is public, or whether it is still the canonical reference.

The source map records the control fields before the content enters the pack. Rendered as a table, the first rows look like this:

This is the same engineering move that appears in prompt lineage and model upgrade gates. Do not trust the final artifact by inspection alone. Record the upstream contract that explains why the artifact is allowed to exist.

Freshness Is a Contract

Agents are very good at making stale documentation sound current. That is the dangerous failure mode. If the pack does not expose freshness, the model has no reason to treat an old migration page differently from a current API reference.

The repo defines freshness classes. Rendered as a table, the policy is easier to scan:

The specific time windows are policy choices. The important part is that each source has a rule. If a source is stale, the correct behavior is not to answer confidently from old material. The correct behavior is to mark uncertainty, remove it from the public index if it can mislead users, and update the pack before relying on it again.

Exclusions First

A documentation pack should not be built by including everything and redacting later. That turns sensitive context into a cleanup problem after it has already entered files, indexes, logs, eval traces, or downstream caches.

The exclusion policy starts with material that must never appear in public packs:

## Never Include in Public Packs

- API keys, access tokens, secrets, passwords, or private certificates
- customer data, support tickets, billing records, or personal data
- unreleased roadmap details
- internal incident reports
- internal deployment procedures
- infrastructure credentials or hostnames that are not public
- private repository URLs
- private Slack, Teams, email, or ticket links
- instructions for bypassing security, rate limits, approvals, or monitoring
- generated summaries that do not cite canonical sources

This is not just security hygiene. It changes answer quality. If excluded material is not available, the agent cannot accidentally treat a private support note, draft roadmap, or stale generated summary as public truth.

Generated Summaries Need Source Links

A generated summary can be useful inside a context pack, especially when the original sources are long. But generated summaries are derivative material. They are not the source of truth.

The repo allows generated content only when it is governed:

Generated summaries can be included only when:

- they cite canonical sources
- they have an owner
- they have a generated-at timestamp
- they are regenerated when sources change
- they are clearly labeled as derivative material

That rule prevents a common context failure: one model summarizes stale docs, another model later treats the summary as authoritative, and nobody can trace the answer back to the original source.

Eval the Pack, Not Just the Model

Teams often evaluate whether a model answered documentation questions correctly. That is useful, but incomplete. You also need to evaluate whether the documentation pack contained the right evidence in the first place.

The repo includes pack-level questions:

## Pack Coverage Questions

1. What is the purpose of this documentation context pack?
2. Which file is the curated public index?
3. Which file defines the source ownership model?
4. Which content is explicitly excluded from public packs?
5. What should happen when a source fails its freshness rule?
6. Is llms-full.txt required by the base llms.txt proposal?
7. Which sources are safe for public agent use?
8. Which sources require authentication before agent use?

These questions are not trying to prove model intelligence. They are testing whether the pack has the evidence needed to answer correctly and safely. If the model cannot answer from the pack, the first fix may be the documentation contract, not the prompt.

Internal Packs Are Different

A public llms.txt file should contain only public material. Internal context packs are a separate deployment problem. They may include runbooks, architecture notes, support procedures, or implementation references, but only behind the same access controls that protect the underlying sources.

The context pack contract makes that explicit:

### Internal Pack

An internal pack may include private runbooks, architectural notes, or support
procedures. It must live behind normal authentication and authorization. It
must not be exposed through a public llms.txt endpoint.

This is where many agent-documentation designs get sloppy. "Useful to the agent" is not the same as "safe to publish." If the content is private for humans, it is private for agents too.

Why This Architecture Works

The pattern works because it separates the documentation surface into responsibilities that can be reviewed independently:

• The public index tells agents where to start
• The expanded pack carries more context only when that is intentional
• The source map records ownership, sensitivity, and freshness
• The freshness policy prevents old docs from sounding current
• The exclusions policy blocks unsafe or unstable material before ingestion
• The evaluation questions test the pack as an artifact
• The internal pack boundary keeps private context behind normal access control

The model still reasons probabilistically over whatever context it receives. The deterministic layer decides what context it is allowed to receive, which sources are current, which sources are excluded, and how the pack is tested.

Next Steps

To extend the operating model, consider:

• Add a validation script that checks every llms.txt link resolves
• Add source-hash recording so expanded packs can detect upstream changes
• Add a CI check that fails when a source has no owner or freshness rule
• Add separate public and internal pack manifests
• Add eval runs that compare answers with and without the context pack
• Add a retrieval adapter that loads only the source classes needed for a task

Those are implementation layers. The important first step is still the same: define the context boundary before automating ingestion.

Final Notes

Agent-readable documentation is not a bigger docs dump. It is a smaller and more explicit contract.

A good llms.txt file gives agents a clean starting point. A production context pack goes further: it records source ownership, freshness, exclusions, internal boundaries, and eval questions. That is the difference between making docs available to agents and making docs reliable enough for agents to use.

Explore the repository at the GitHub repository.

See you in the next issue.

Stay curious.