In the previous post we explained what Agent Readiness is and why our site jumped from 20 to 93 in a day. This time we open the toolbox. Every file an AI agent looks for on a site, what goes in it, and what it means in plain language. At the end there is a checklist for anyone with a WordPress site.
What you will learn from this post
- What to write in robots.txt so AI bots understand they are welcome (or not)
- What Content Signals are, what the three words there mean, and what we chose to declare and why
- The difference between the sitemap, llms.txt and Link headers, and why you need all three
- What a markdown version of a page looks like, and the numbers we measured
- What lives in the
.well-knownfolder, and why it is less scary than it sounds
Who this post is for: site owners who want to understand exactly what their maintainer needs to do (and check that they did it), and developers who want the list in one place. If you only want the general idea, the previous post is enough.
File 1: robots.txt, the veteran
robots.txt has existed since 1994. It is a text file at the root of the site that tells robots what they may crawl and what they may not. Almost any site that has had SEO work knows it.
What is new: AI bots expect lines addressed to them by name. Not only User-agent: * (everyone), but:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /Why does this matter if “everyone allowed” already covers them? Because Cloudflare’s validators say explicitly that a wildcard alone “is not sufficient” as an AI policy statement. And because one day you may want to distinguish: yes to ChatGPT, no to a particular bot. The main names today: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Google, for training Gemini), PerplexityBot, CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, Applebot-Extended, meta-externalagent.
File 1, one more line: Content Signals
This is a new addition to the same file. One line:
Content-Signal: search=yes, ai-input=yes, ai-train=yesThree words, three declarations:
| Signal | What it means | Example |
|---|---|---|
search | may index the content and show a link to it | an ordinary search result |
ai-input | may feed the content to a model while it answers, i.e. quote it | Google AI Overviews, a ChatGPT answer with a source |
ai-train | may train models on the content | the model “remembers” you even without searching |
What we declared: yes, yes, yes. And that is a business decision, not a technical one. Our site is marketing. The content is the advertisement. Being quoted in an AI answer is exactly the product we sell (that is GEO). And being in the training data is the only way a model answers “where do I get a fast website in Israel” from memory, even when the user is not searching the web.
The default Cloudflare suggests is different: search=yes, ai-train=no. That is a sensible default for a site that sells content: a newspaper, paid guides, a creator selling the work itself. There, a model that learned the content replaces the visit. For a business that sells a service, a model that learned the content brings the visit.
When no makes sense: content that is sold, original creative work, user data, or a matter of principle. One of our clients, an artist, had to decide this herself because it is her own artwork. She chose yes.
⚠️ What you need to understand: Content Signals are a declaration, not a wall. A well-behaved bot honours them. A badly behaved one does not. Anyone who wants a real technical block needs a network-level tool (Cloudflare has AI Crawl Control), and that is an entirely different discussion.
File 2: sitemap.xml, the complete list
The sitemap is a list of every address on the site. An agent (and Google) finds it through a line in robots.txt: Sitemap: https://.../sitemap-index.xml.
The only thing I will add here, because it burns people: the modification date (lastmod) must be real. A sitemap that marks every page as “updated today” on every build teaches Google that the signal is noise, and it stops reading it. On our site only posts get a date, from their real publication date; static pages get no date at all.
File 3: llms.txt, the short guide
If the sitemap says “here are 140 addresses”, llms.txt says “here is who we are, and here are the important pages in the order you should understand them”. A free-text file, at the site root, in markdown format:
# Prime Stack - Premium Tech. Fast Results.
> A digital agency building fast websites with Astro...
## Key pages
- [Pricing](/pricing/): plans and prices
- [Upgrade an existing site](/upgrade/): ...llms.txt is a community proposal, not an official standard, and not all the big engines have committed to reading it. Cloudflare’s scanner does not check it at all. We still recommend it: it is cheap, harmless, and some engines do read it. A tip from experience: do not write it by hand. Ours is generated automatically from the page list on every build, because the first hand-written version went stale within a week: we added a pricing page and the file did not know.
File 4: Link headers, the addresses that arrive with the page
This is not a file but a “header” the server sends along with the homepage. The visitor does not see it. An agent does:
Link: </.well-known/api-catalog>; rel="api-catalog", </llms.txt>; rel="describedby"In words: “I have an API catalog at this address, and a description of myself at that one”. The agent does not have to guess where to look. An old standard (RFC 8288) in a new use.
File 5: a markdown version of every page
This is, in my view, the most important item on the whole list, and the only one that requires real work.
A web page is an HTML file. Inside it: a menu, a header, a footer, analytics scripts, a cookie banner, a chat widget, styling, and somewhere inside all of that, the text. An AI agent pays for every word it reads (tokens), and has a limited budget per page. When it hits 150 kilobytes of HTML, it may read part and skip, or read the menu and miss the price.
The new standard (Cloudflare calls it “Markdown for Agents”) says: when an agent sends a request with the header Accept: text/markdown, the server returns a clean text version of the same page. A regular visitor, who does not ask for it, keeps getting the normal HTML. Same address, two versions.
What we measured on our site after building it:
| Page | HTML | markdown |
|---|---|---|
| Homepage | 158 KB | 12 KB |
| Pricing | 93 KB | 5.5 KB |
Between 6% and 8% of the size. Cloudflare reports savings of up to 80% in tokens on its own sites; ours is more, because brochure sites carry a lot of design relative to text.
What does it look like? This is how our pricing page begins in its markdown version:
---
title: "Pricing | Prime Stack"
url: https://primestack.co.il/en/pricing/
lang: en
---
# Pricing
Transparent prices, no surprises - the price you see is the price you pay.
## New website
### Long-form brochure site (1Pager)
...Content only. Headings, paragraphs, tables, links. No menu, no footer, no script.
How it is built: on our site, a script runs on every build, takes the main part of each page (<main>), strips anything that is navigation or code, and converts it to text. The server checks whether the request asks for markdown, and if so serves the twin file. Two details you must not miss: (1) a Vary: Accept header, which tells caches “this response changes with what was asked for”, otherwise a visitor may receive markdown; (2) if the site has any mechanism that blocks pages (ours has a Shabbat gate), it must let these requests through.
⚠️ Cloudflare has a built-in switch for this, no code, but only from the Pro plan up. On the free plan you build it yourself, as we did.
Folder 6: .well-known, the machines’ cupboard
.well-known is a standard folder (with a leading dot) where machines look for files by a name known in advance. You already use it without knowing: SSL certificate validation, Apple’s verification for Apple Pay, and more. Now it has files for agents too:
api-catalog- the list of programming interfaces (APIs) the site exposes, per RFC 9727. We have one: the speed test. If you have no API, you can list llms.txt as the “service description”.agent-skills/index.json- an index of “operating manual” files for the agent, each with a digital signature (sha256) proving the file has not changed. Ours say, for instance: “quote prices only from the pricing page, they are final; the site closes on Shabbat, do not promise availability”.ai-catalog.json- a general catalog of all capabilities (the ARD standard, a draft). Points at the other files.openapi.json- a technical description of the API, for those who have one.
And one file outside the folder, at the root: auth.md, which explains to the agent in words whether and how it needs to log in. Ours says: no need. Everything is open, and the speed test API is limited to 10 checks a day per address.
Two things not on the list, on purpose
DNS-AID - a DNS record announcing an entry point for agents. We added it because it is a single record, but it is a very early draft and I would not send anyone to do it for the outcome. For the score only.
WebMCP - a few lines of JavaScript on the page telling an agent’s browser “here are actions you can take here”. Ours: site search, reading a page as markdown, opening the speed test. That already spills into the next post, about MCP, because that is where the site stops being a page and starts being a tool.
Checklist for WordPress site owners
What to ask your maintainer for, in order of cost:
- In robots.txt: a
Content-Signalline (decide first what to declare) and explicit blocks for AI bots. Through the SEO plugin. - A sitemap with real modification dates, and a Sitemap line in robots.txt. Most SEO plugins do this.
- An llms.txt file at the site root: a paragraph about the business and a list of the 8-12 important pages. You can start by hand, and remember to update it.
- A markdown version of pages: ask whether there is a suitable plugin for your WordPress version, or whether the site is on Cloudflare Pro (built-in switch). If not, it is development work.
- A
.well-knownfolder with api-catalog and agent-skills: development work, about an hour for someone who has done it once. - Run isitagentready.com before and after, and keep a screenshot.
Doing the first three gets you to Level 2. With markdown, Level 3. With .well-known, Level 4.
Why it took us one evening: a word about Astro
Everything described here happened on a site built with Astro, the technology we build all our sites with. It is worth explaining why that matters, because the difference was not in knowledge but in the tool.
An Astro site is a site written in code and rebuilt into ready files every time something changes. There is no server assembling pages on the fly, no database, and no plugins each of which is a world of its own. Every page, menu and piece of content sits in files you can read, change and run a script over.
What that gave us here, in practice:
- A markdown version of every page is one script that runs at build time, walks the 140 finished pages and writes a twin for each. We wrote it once. From now on every new page gets a twin without anyone remembering to do it.
- The digest signature of the agent instructions is computed from the file itself at the same step, so it cannot go stale.
- The llms.txt file is generated from the real page list, not from a file someone typed a month ago.
- The server logic (answering markdown requests, sending Link headers, exempting the files from the Shabbat gate) is a few dozen lines in one file that already existed.
On a WordPress site the same work looks different. robots.txt and Content Signals go through the SEO plugin, and that is fine. But a markdown version of every page needs a dedicated plugin (they are only starting to appear) or server code; .well-known files are uploaded by hand over FTP or configured in .htaccess; and the digest of the agent instructions has to be recomputed by hand every time someone touches the file. Each item is a separate task, with a separate plugin, that also has to be updated separately. Not impossible, just more work and more points of failure. We compared the two approaches in depth in WordPress, Wix or custom code: an honest comparison.
And it is part of something broader we see again and again: when the web changes its rules, a site built in code adapts with a script, and a site built from plugins waits for a plugin. That is how it went with speed, with accessibility, and now with AI agents. If you want to understand why we moved to Astro after hundreds of WordPress sites, we wrote about it in why I moved from WordPress to Astro, and if you have an existing site and want to know what the move involves, that is the upgrade page.
In summary
All the “files for agents” are simply written answers to the questions an agent asks: what am I allowed (robots.txt and Content Signals), what is here (sitemap, llms.txt, Link headers), how do I read it cheaply (markdown), and what can I operate (.well-known). Most are text files of a few lines. One, the markdown, needs a day of work and is worth it.
The only decision that is truly yours, and not your maintainer’s, is what to declare in Content Signals. If your site is marketing, the answer is almost always “yes, yes, yes”. If the content itself is the product, think for a moment first.
In the next and last post in the series: MCP, WebMCP and A2A. What happens when an agent does not just read the site but operates it, what that can give a business, and what we are thinking of building from it.