Denshin / Blog / AI

llms.txt and AI crawlers: what to put on your site

A practical guide to the machine readable layer of a website: what llms.txt is and why it is a proposal rather than a standard, how to tell training crawlers apart from the retrieval fetchers that power AI citations, and the robots, sitemap, canonical and rendering work that definitely matters.

Denshin Team · Product & Engineering · 27 August 2026 · 7 min read

Somewhere between "we should do something about AI" and an actual work ticket sits a small pile of files at the root of your domain: robots.txt, sitemap.xml, and now maybe llms.txt. This is the machine readable layer of your site, and it is one of the few pieces of AI related work that is cheap, reversible, and mostly under your control. Here is what each file is for, what is a real standard and what is a proposal, and the one configuration mistake that quietly costs sites their visibility in AI answers.

What is llms.txt?

llms.txt is a proposed convention for a markdown file at the root of your domain, at /llms.txt, that gives a language model a curated map of your site: a short description of what the site is, and a list of links to your most useful pages in a clean, readable form. The idea is that a model with a limited context window should not have to guess which of your two thousand URLs matter.

Be honest about its status. As of writing in August 2026, llms.txt is a community proposal with uneven adoption, not a ratified standard and not something any major vendor has publicly committed to treating as a ranking input. Treat it as cheap insurance: it costs an afternoon, it cannot hurt you, and it doubles as a genuinely useful index for your own team. Do not treat it as a lever, and be suspicious of anyone selling llms.txt optimisation as a service.

What to put in it

Keep it short and factual. A workable shape:

# Denshin

> A software studio in India building web products on React and AWS serverless.

## Services
- [What we build](https://www.denshin.in/services): scope of the studio's work

## Writing
- [Pre-launch checklist](https://www.denshin.in/blog/the-pre-launch-checklist-we-run-before-shipping): what we verify before a site goes live
- [Serverless cost math](https://www.denshin.in/blog/serverless-math-when-lambda-is-cheaper-than-a-server): when Lambda is cheaper than a server

## Contact
- [Contact](https://www.denshin.in/contact): how to reach the team

Rules of thumb: absolute URLs, one line of context per link so the entry is useful without a fetch, no marketing copy, and no pages you would not want quoted. Some sites also publish clean markdown versions of individual pages at page-url.md. That is a nice touch if your build can generate it, and a maintenance burden if it cannot. Do not hand maintain a second copy of your content, because the version that drifts is always the one a machine reads.

The file that definitely matters: robots.txt

robots.txt is a real, long standing convention that well behaved crawlers respect. It is where you express what you want fetched, and it is where most of the damage gets done, because the AI era added a category of crawler that most teams do not distinguish.

Training crawlers versus retrieval fetchers

There are broadly three kinds of automated fetch you should think about separately.

Type	What it does	If you block it
Training crawler	Collects pages into a corpus used to train future models	Your content is less likely to be absorbed into model weights. No direct effect on today's answers
Search or index crawler	Builds the index an AI answer engine retrieves from	You become uncitable in that product. This is the expensive mistake
User triggered fetcher	Fetches a specific URL because a person asked the assistant about it right now	Users who paste your link get "I cannot access that page"

Several vendors run separate user agents for these roles, and some run only one. The names change, new ones appear, and behaviour is documented by each vendor rather than by a shared spec. As of August 2026 the commonly seen agents include OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot family, PerplexityBot, Applebot and Applebot-Extended, Google-Extended as a training preference token distinct from Googlebot, Bingbot, and CCBot for Common Crawl. Verify every one of these against the current vendor documentation before you paste it into a config. A stale user agent list is worse than none, because it gives you false confidence that a decision you made two years ago is still in force.

A decision you should make deliberately

The default we recommend for a marketing site or a blog: allow retrieval and search crawlers, allow user triggered fetchers, and decide on training crawlers according to how you feel about your content in a corpus. Publishers with paid content, or anyone whose text is the product, reasonably block training. A studio that wants to be found and cited has very little to gain by blocking anything.

What almost nobody should do is block everything with an AI sounding name. That is how a site becomes invisible to AI search citation while its owner believes they have taken a principled stand on training data.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# Example only. Verify current agent names and behaviour
# against each vendor's documentation before shipping.
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://www.denshin.in/sitemap.xml

Two caveats. robots.txt is a request, not an enforcement mechanism, and it only binds crawlers that choose to honour it. If you need something not fetched, put it behind authentication. And blocking a path in robots.txt does not remove it from an existing index, it only stops future fetches.

The rest of the machine readable layer

These are not AI specific. They are the same things that made a site legible to search engines, and they carry more weight now, not less.

Sitemap. Generated at build time, listing only canonical, indexable URLs, with honest lastmod values. A sitemap full of stale dates trains everyone to ignore your dates.
Canonical URLs. One canonical per page, self referencing on the canonical version. This is how you stop the same article being treated as three competing sources.
Clean semantic HTML. Real headings in order, <ul> for lists, <table> for tabular data, <time datetime="..."> for dates, alt text on images that carry meaning. Chunkers and extractors follow structure. A page built out of nested <div> elements gives them nothing to follow.
JSON-LD structured data. Organization for the site, Article or BlogPosting for posts with datePublished and dateModified, BreadcrumbList for hierarchy. Describe what is actually on the page, and validate it.
A visible publish and update date. In the HTML, not only in the metadata. Freshness is one of the few signals a synthesiser can check cheaply.

The client side rendering trap

You can get every file above right and still be invisible, because the crawler fetched your HTML and found an empty <div id="root"></div>. Rendering support across AI fetchers is inconsistent and undocumented. Some execute JavaScript, some fetch once and parse what they get.

Test it the direct way, with no tooling and no vendor dashboard:

curl -sL https://www.denshin.in/blog/some-post | grep -c "a sentence from your article"

If that returns zero, your content does not exist as far as a plain fetcher is concerned. The fix is to server render or pre-render the public pages, which for a static marketing site or blog is a build step, not an architecture change. Keep the single page app for the authenticated parts of the product. We laid out that split in S3 and CloudFront for an admin heavy SPA, and it holds up here.

A worked example: a small site's setup

Say you run a ten page site with a blog. The complete job, start to finish:

Generate sitemap.xml at build time from your route manifest. Exclude anything noindex.
Write robots.txt by hand. Allow everything public, disallow /admin and any API path, reference the sitemap, and add per agent rules only if you have actually decided something.
Add a self referencing <link rel="canonical"> to every page template.
Add Organization JSON-LD to the layout and BlogPosting JSON-LD to the post template, populated from the same data the page renders. Never a hand written duplicate.
Pre-render every public route to static HTML.
Write /llms.txt with your ten to twenty most useful URLs and one line each. Add it to the same build script so it is regenerated, not forgotten.
Curl three pages and confirm the text is there. Fetch /robots.txt and /llms.txt in a browser and confirm they return plain text, not your app's 404 page. This last check catches more errors than anything else on the list.

That is roughly a day of work on an existing site, and most of it belongs in the checklist you run before shipping anyway.

What to do next

Order of value, highest first: make sure the text is in the HTML, fix canonicals and the sitemap, add honest structured data, make a deliberate decision about training versus retrieval crawlers and write it into robots.txt with a dated comment explaining the decision, and then, if you have an hour left, publish an llms.txt. Re-read your crawler rules against vendor documentation every couple of quarters, because that list is the part of this that goes stale.

If you want someone to audit what a crawler actually sees on your site, or to build the pre-rendering and structured data into the next version of it, get in touch.

Tags: llms.txt, AI Crawlers, Technical SEO, AI Search, Structured Data

All posts · Work with Denshin