What is llms.txt? It is a plain markdown file at yoursite.com/llms.txt that gives AI tools a short summary of your site and a list of its key pages. Jeremy Howard of Answer.AI proposed it on 3 September 2024. Google says Search ignores it, and none of OpenAI, Anthropic or Perplexity says its crawlers read it, so treat it as optional housekeeping.
We add llms.txt to client sites in most of our GEO sprints, and we serve one ourselves at citevantage.com/llms.txt. This guide covers what the spec requires, what each engine’s own documentation says, what we found when we graded our own file on 7 November 2026, and how to build one in about half an hour. Every source below was opened on 7 November 2026, and the figures are quoted as each source gives them.
What is llms.txt exactly?
llms.txt is a text file written in markdown, which formats text with plain characters: a # makes a heading, a > makes a quote, and [text](url) makes a link. AI models read it well because it is clean text with clear structure. It sits in your site’s root directory, the top level of your website where robots.txt lives, so on example.com it loads at example.com/llms.txt.
The idea behind it answers a real problem. Web pages carry navigation menus, cookie banners, scripts and ads, so a model fetching one has to dig the content out of that noise within a limited context window (a cap on how much text it can read at once). llms.txt hands the model a clean version: here is what this site is, here are the pages that matter, go read these.
What does the llms.txt spec require?
Only the H1 is required. The spec at llmstxt.org lists itself as version 2, published 3 September 2024 and modified 10 August 2026, and describes the file in this order:
| Part | Required? | What the spec says |
|---|---|---|
| H1 with the site or project name | Yes | The spec says: “This is the only required section.” |
| Blockquote summary | No | A short summary with the key context |
| Detail sections | No | More background, in any markdown except headings |
| H2 sections of links | No | Each item is a [name](url) link, optionally followed by a colon and notes |
| An H2 called “Optional” | No | By convention, secondary links an agent can skip when it needs a shorter context |
| Location | The site root, or any path within it to cover the pages under that path |
The spec also suggests serving a clean markdown copy of each page at the same URL with .md added, and pointing to it with a rel="alternate" type="text/markdown" link. We have not tested whether any engine uses those copies. The rest is plain: no code, no special syntax and no tooling.
Who created llms.txt and why?
The spec lists Jeremy Howard of Answer.AI as its author and gives 3 September 2024 as the date the proposal was published.
It spread fastest in the developer world. Mintlify, a documentation platform, announced on 20 November 2024 that it would generate llms.txt and llms-full.txt for the docs sites it hosts, so its customers gained the file without writing it. On 7 November 2026 we loaded a live llms.txt from the developer docs of Anthropic, OpenAI, Perplexity and Cloudflare.
Outside developer docs, the file is rarer. SE Ranking found it on 10.13% of roughly 300,000 domains it sampled in a study published on 15 October 2026, and Ahrefs found it on 28% of the 137,210 domains in its sample, which it calls an upper bound because its customers skew technical.
What does llms.txt actually do?
On its own, the file sits at your root and waits to be read, and no engine is obliged to fetch it. When something does fetch it, the use cases are these:
- A person or tool is pointed at it. Some developer tools, coding assistants and AI agents let a user supply an llms.txt URL to load a site’s content as context. Ahrefs’ authors say AI tools fetch the file “when a link, an index, or a user instruction tells them it exists”, and that none of the AI bot requests it measured went looking for a missing file.
- A browsing audit checks it. Chrome’s Lighthouse has an llms.txt check in its agentic browsing audits. Per its documentation, a server error while fetching the file fails the audit, and a 404 is marked not applicable because the file is “optional at the moment”.
- You curate your own summary. Without the file, a model samples whatever it happens to crawl. With it, your best area guides, agent pages and policies sit in one place with descriptions you wrote.
It blocks no crawler, controls no training and adds no structured data.
How is llms.txt different from robots.txt and sitemap.xml?
These three files all live at your root and all talk to machines, so they get confused constantly. They do three different jobs: robots.txt grants or denies access, sitemap.xml lists every URL you want crawled, and llms.txt proposes which pages matter and what they mean. Only the first two are established standards.
Pro tip: Write every entry as a real markdown link,
- [Page name](https://example.com/page): what it covers. We shipped a version with bare URLs and Google PageSpeed reported the file as containing no links at all, because the parser reads markdown link syntax rather than raw text.
| Aspect | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Job | Blocks or allows crawlers | Lists every URL for indexing | Curates key content for AI |
| Says | ”Do not go here" | "Here is everything" | "Here is what matters, and why” |
| Format | Plain text rules | XML | Markdown |
| Audience | All crawlers (Googlebot, GPTBot, etc.) | Search engine indexers | AI models and AI tools |
| Curated? | No | No | Yes, hand-picked pages with descriptions |
| Standard status | Established since 1994, universally respected | Established since 2005, universally used | Proposed 2024, no official AI adoption |
| Required? | Effectively yes | Strongly recommended | Optional |
llms.txt is a poor tool for permissions, so stopping GPTBot or ClaudeBot is a robots.txt job. If you want AI citations, also make sure you are not blocking those crawlers by accident, because a firewall or CDN can refuse an AI crawler without the site owner knowing, as our own host did to GPTBot. That access check is part of generative engine optimization, the wider discipline this file belongs to.
What does a real llms.txt look like, and how did ours grade?
The fastest way to learn the format is to read a live file, so here are the opening lines and one section of ours, copied on 7 November 2026 with line breaks added so they fit the page (the live file keeps each entry on one line). The full file is at citevantage.com/llms.txt.
# CiteVantage
> AI SEO and GEO agency for real estate. CiteVantage gets real
> estate agents, teams and brokerages named and cited by ChatGPT,
> Google AI Overviews, Google AI Mode, Gemini, Perplexity and
> Copilot when buyers and sellers ask which agent to work with.
## Reference
- [Glossary](https://citevantage.com/glossary/): GEO/AEO terms defined.
- [FAQ](https://citevantage.com/faq/):
common AI visibility questions answered.
- [Can AI crawlers reach your site](https://citevantage.com/blog/can-ai-crawlers-reach-your-site/):
how to test crawler access yourself.Reading it line by line:
- The H1 is the only line the spec requires, and ours is the brand name alone.
- The blockquote is the summary. It names the service, the vertical and the engines in one sentence, so a tool that reads only that line can still describe us.
- Each list item is a markdown link, a colon and a short description. An entry without the description is still valid, but it gives a reader no reason to follow it.
We then graded the whole file the way we grade a client’s, by fetching it and requesting every link. This is what the check returned on 7 November 2026:
| Check | Result for citevantage.com/llms.txt |
|---|---|
| Served as | HTTP 200, text/plain, 19,998 bytes, 139 lines |
| Structure | One H1, one blockquote summary, 12 H2 sections |
| Links | 90 entries pointing to 88 distinct URLs, none written as a bare URL |
| Do the links resolve? | All 84 distinct citevantage.com URLs returned 200 with no redirect |
| Descriptions | 54 of 90 entries have one, and 36 are bare links |
| Optional section | None |
| Date line | Reads “Last updated: 2026-09-24”, although the file lists a study published in November 2026 |
| llms-full.txt | HTTP 200, text/plain, 997,284 bytes |
The file is valid and everything it points to resolves, which is where many files fail. It also runs well past our own rule of thumb of 10 to 30 links (the spec sets no limit), leaves 36 entries undescribed and carries a date line that is out of date. We fixed three of those the same day: every entry now has a description, the date line reads 7 November 2026, and the YouTube link sits under an Optional heading. We added the new pages we had published since September, so the file now holds 100 links, and we have not trimmed it. We have no evidence that any of these changes whether an engine reads the file.
If you are starting from nothing, this is a template for a small brokerage. Every name and URL in it is a placeholder:
# Your Brokerage Name
> Two to four sentences: what you do, where you work and one fact that proves it.
## Areas we serve
- [Neighbourhood guide](https://example.com/areas/neighbourhood): schools, price range and days on market, updated monthly.
## Guides and market reports
- [Market report](https://example.com/market-report): median price and inventory for the month.
- [How to sell a home here](https://example.com/guides/selling): timeline, costs and a preparation checklist.
## Company
- [Our agents](https://example.com/agents): names, license numbers and service areas.
## Optional
- [Press](https://example.com/press): local news coverage and community sponsorships.Four habits separate a useful file from a pile of links:
- The summary answers “who is this?” in one breath. What you do, where you work and what makes you credible.
- Every description carries a specific. A price range, a count or a date gives a reader a reason to click.
- The guides lead with question-shaped content. The same answer-first pages that earn citations on their own merit, as we cover in how to get cited by ChatGPT.
- The file stays short. Ten to thirty links is our rule of thumb for a small site, because a dump of every URL gives a reader nothing to choose between.
What is llms-full.txt?
llms-full.txt is the expanded companion file. Where llms.txt is an index (summary plus links), llms-full.txt contains the full text of your key pages, converted to markdown, in one large file. The point is to let an AI read a site’s substance in a single fetch, with no crawling at all.
Documentation sites use it heavily. When we fetched them on 7 November 2026, Anthropic’s llms-full.txt was 50,342,554 bytes and ours was 997,284. We have not tested which tools can load either one whole.
For a typical agent, brokerage or service site it is optional. Listings change too often to maintain a full-text mirror by hand, so start with llms.txt and add the full version only if you have stable guide content and a way to regenerate it.
How do you create an llms.txt file? (Step by step)
The whole process takes 15 to 30 minutes for a small site: pick the pages that answer real buyer questions, write one honest sentence about each, and upload the file to your root.
- Pick your pages. List the 10 to 30 pages that best represent your business: area pages, best guides and market reports, agent pages, your about page and policy pages. If a page would embarrass you as the only thing an AI ever read about you, leave it out.
- Write the H1 and summary. Make the first line
# Your Brand Name, then add a blockquote (a line starting with>) of two to four sentences saying what you do, which areas you serve and one credibility fact, written so it could be quoted word for word. - Build the sections. Add H2 headings (
## Areas we serve,## Guides,## Company), each followed by a markdown link list in the form- [Page Title](full URL): one-line description with a specific fact.Use full absolute URLs, not relative paths. - Add an Optional section if needed. Skippable pages go under
## Optional, which the spec treats as content an agent can drop when room is tight. - Save and upload it. Save plain UTF-8 text under exactly the filename llms.txt, all lowercase, and place it in the same folder as your homepage, next to robots.txt. On WordPress, use your host’s file manager or a plugin that generates the file.
- Verify it loads and every link resolves. Visit yoursite.com/llms.txt in a browser and you should see raw markdown as plain text. A 404 means the file is in the wrong folder. Then request each linked URL once and fix anything that errors or redirects, which is the check our own file passed on 7 November 2026.
- Put a review date on your calendar. Once a quarter, update the links and facts, and update the date line if you keep one. A stale llms.txt pointing at dead pages is worse than none.
Free generators, including open-source ones such as Firecrawl’s, can draft the file from your sitemap, but edit the output, because the value of the file is curation and a machine-generated dump of every URL is not curated.
How do you set up llms.txt on WordPress or a hosted agent website?
WordPress gives you root access, and an IDX plugin does not change where root files live, so step 5 above applies unchanged. Hosted agent and brokerage platforms may not let you upload arbitrary files to the site’s root directory, and a file uploaded through a media library can land on a CDN under a different URL, where it will not load at yoursite.com/llms.txt. Some platforms may generate the file for you. Your realistic options:
- Check whether it already exists. Platforms keep adding AI-related files natively, so visit yoursite.com/llms.txt before building anything.
- Ask your platform. Its support team can tell you whether it serves the file, or whether a template or custom page can be published at that exact path.
- Skip it for now. Honestly a defensible choice on a locked-down platform. The engineering workaround costs more than the unproven benefit, and answer-first area pages, structured data and crawler access matter far more. Structural and on-page work, with no root file involved, is what moved the numbers in our Bright Realty International and RIPT Apparel case studies.
Whatever you do, do not pay a meaningful monthly fee just to serve this one static file.
Does llms.txt actually work in 2026?
No provider has said it does. Google says Search ignores the file, the three AI companies we checked document their crawlers without mentioning it, and the log studies show it going largely unrequested. We found no test showing that adding one changes whether you get cited.
No major AI platform has committed to reading llms.txt.
Here is what each source states, read on 7 November 2026.
| Source | What it states |
|---|---|
| Google Search Central, AI optimization guide | The guide says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.” Creating them for other services is fine, and doing so “will neither harm nor help your site’s visibility or rankings in Google Search.” |
| OpenAI crawler documentation | Describes OAI-SearchBot, GPTBot and ChatGPT-User and how robots.txt applies to each. llms.txt appears only as a link to OpenAI’s own docs index. |
| Anthropic crawler help article | Describes ClaudeBot, Claude-User and Claude-SearchBot and how robots.txt applies to each. No mention of llms.txt. |
| Perplexity crawler documentation | Describes PerplexityBot and Perplexity-User. llms.txt appears only as a pointer to Perplexity’s own docs index. |
| Chrome Lighthouse | Flags a server error when fetching the file, and treats a 404 as not applicable because the file is “optional at the moment”. |
We did not check Microsoft’s or Meta’s documentation. The log evidence is thinner than the policy statements, so it helps to see the three studies side by side:
| Study | Sample | What it found |
|---|---|---|
| Ahrefs, 15 June 2026 | 137,210 domains, May 2026 | 28% had a file, an upper bound. About 97% of those files received no requests. AI retrieval bots made up 1.1% of requests to the files that got any. The authors say the file “won’t help your AI search visibility today.” |
| EZY.ai, 12 weeks to 19 July 2026 | 83 small-business sites with the file live | Fetches of robots.txt against llms.txt: OpenAI 3,990 against 7, ClaudeBot 3,120 against 9, PerplexityBot 775 against 0, Googlebot 5,125 against 67, Meta-ExternalAgent 172 against 193. EZY.ai sells a tool that generates the file. |
| SE Ranking, 15 October 2026 | About 300,000 domains | 10.13% had the file. A model predicting citations found no correlation with it, and the authors say it does not appear to affect citations “at least not yet”. |
Two limits apply to all of it. A fetch does not mean the content was read, and Ahrefs calls every figure in its study “a ceiling on actual llms.txt consumption.” And one practitioner reported in July 2025 that OpenAI was requesting the file on a few of their sites, a single log screenshot that OpenAI’s own documentation does not confirm. John Mueller of Google said in April 2025 that “AFAIK none of the AI services have said they’re using LLMs.TXT” and compared the file to the keywords meta tag. The newer Google documentation above is the better guide.
And yet companies keep publishing it. Anthropic, OpenAI, Perplexity and Cloudflare each serve one for their own docs. There are three reasons, and they are the reasons we still ship it:
- It costs almost nothing. Twenty minutes and no known penalty. The only upkeep is a review each quarter.
- It works for pull-based use. When a person or an AI agent deliberately loads your llms.txt into a tool, the file does its job today. That use case is real, just niche.
- It is a hedge on adoption. If an engine starts reading it, sites with clean files are ready on day one. robots.txt itself began as an informal convention.
Our verdict is that llms.txt is cheap insurance with no measured citation benefit. Structured data is not a single lever either, and our guide on which schema types matter for AI search sorts the types that verify facts AI cannot guess from the ones that are just hygiene. If a vendor is selling llms.txt as the secret to AI rankings, close the tab.
Should you add llms.txt to your site?
Yes, once the higher-leverage work is done or under way. It costs half an hour, it cannot hurt you and it is trivially reversible, which is a different argument from saying it works. Our priority order:
- Answer-first content that resolves real buyer questions.
- Crawler access verified for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in robots.txt and at the firewall.
- Fresh, regularly updated key pages.
- Structured data plus the same facts in visible text.
- Earned third-party mentions.
- Then llms.txt, in the 20 spare minutes.
The file is a finishing touch on a site that is already extractable and crawlable, and a distraction on one that is not. Every full GEO engagement we run through our services includes it, in exactly that position. For the work behind items 1 to 5, start with how to rank in ChatGPT.
Should a real estate agent or brokerage publish llms.txt?
A real estate agent or brokerage can publish llms.txt in twenty minutes, but it belongs after crawler access and answer-first area pages. In our 500-query Houston study, a firm’s own domain appeared in the citations of only 8.4% to 10.4% of the AI answers that named it, so a file on that domain is a small lever working on a small share of what the engines cite. Those answers come from our own test runs, which are directional and are not a market share.
Houston study
to 10.4% of the AI answers that named a firm cited that firm’s own domain, so a file on it is a small lever.
Photo: Curtis Adams, Pexels
The bigger risk on an agent site is a crawler that never arrives. When we tested our own host in September 2026, the full GPTBot/1.2 user agent got through on 0 of 8 cache-miss requests while every other identity we tried, from OAI-SearchBot to an ordinary Chrome browser, got through on 8 of 8. No robots.txt setting or llms.txt file on the site could have shown that, and the evidence is in the GPTBot case study. If your site sits behind a host or platform you did not configure yourself, our guide to checking whether AI crawlers can reach your site is the test to run before writing a single line of the file.
Once access is confirmed, list the area pages, market reports and agent pages that answer real buyer and seller questions, using the template above, and keep the file current as agents join or leave. Portals and agent referral networks took 43.6% of the citations in the Houston study, which is where our GEO for real estate work concentrates, and the full playbook for agents is in AI search for real estate agents.
Find out what AI engines actually say about you
Before adding any file, find out where you stand. Our free AI Visibility Audit runs your brand through ChatGPT, Perplexity, Gemini and Copilot against real buyer questions in your category, scores you 0 to 100, and shows exactly who gets cited instead of you. For agents, teams and brokerages, the AI visibility audit for real estate runs those questions area by area.
In our 500-query Houston study, an estimated 94.0% of the Houston Association of Realtors’ 50,000-plus members were never named in 2,000 AI answers. The audit shows what would actually change that for your business.
Get your free AI Visibility Audit. Takes two minutes to request, and you get the citation gap map whether you work with us or not.
Frequently asked questions
What is llms.txt in simple terms?
Think of it as a short markdown brochure for AI: a file at yoursite.com/llms.txt that summarises your site and links to the pages that matter most.
Jeremy Howard of Answer.AI proposed it on September 3, 2024, because web pages are cluttered with menus, scripts and cookie banners while a model can only read so much text at once. The file hands the model a clean version instead, with one H1, a short summary and organised link lists. It is still a proposal, and Google says its Search ignores the file.
Is llms.txt the same as robots.txt?
No, the two files do opposite jobs: robots.txt tells crawlers where they may not go, while llms.txt blocks nothing and simply points AI at your best pages.
robots.txt has been the access rulebook since 1994 and every major crawler respects it. llms.txt, proposed in 2024, is a curated guide with no enforcement behind it, so stopping GPTBot or ClaudeBot is still a robots.txt job. Most sites should run both, and while you are in robots.txt it is worth confirming you are not blocking AI crawlers by accident.
Do AI engines like ChatGPT actually read llms.txt?
No engine we checked says so: as of 7 November 2026, OpenAI, Anthropic and Perplexity document their crawlers without mentioning llms.txt, and Google says Search ignores it.
Google's guide to AI features says creating such files will 'neither harm nor help' your visibility in Google Search. Server logs point the same way: Ahrefs found that about 97% of the llms.txt files on its 137,210 sampled domains received no requests in May 2026, and in a 12-week panel of 83 sites run by EZY.ai the OpenAI crawlers fetched the file 7 times, ClaudeBot 9 times and PerplexityBot never. SE Ranking, testing about 300,000 domains, found no correlation between having the file and being cited. None of that proves the file is useless forever, but it gives no reason to expect a citation lift.
How do I create an llms.txt file?
Write a short markdown file, save it as llms.txt and upload it to your site root so it loads at yoursite.com/llms.txt.
For a small site the job takes 15 to 30 minutes, and the step-by-step section in this guide walks through every line. The order we follow:
- Pick the 10 to 30 pages that best represent the business.
- One H1 with your site name, then a blockquote summary of two to four sentences.
- Group the pages under H2 headings, each written as a real markdown link with a one-line description.
- Plain UTF-8 text, saved under an all-lowercase filename.
- A browser check confirms it loads, since a 404 means the file sits in the wrong folder.
How do I check that my llms.txt is valid?
Fetch it and confirm four things: a 200 status, a text/plain content type, an H1 as the first line, and links that each resolve without a redirect.
The spec itself only requires the H1, so there is no official validator to pass or fail. We run the check by hand on every client file:
- Open yoursite.com/llms.txt and confirm raw markdown shows, with no HTML wrapper.
- Check that every entry is a markdown link, because bare URLs are not read as links.
- Request each linked URL and flag anything that returns an error or redirects.
- Look at the date line, if you have one, and the descriptions, since a stale file points readers at dead ends.
Do I need llms.txt for my real estate website?
Optional, and low on the priority list for most agent and brokerage sites.
A hosted website platform may already generate the file for you, so check yoursite.com/llms.txt first to see whether one already exists and what it says. If answer-first area pages, crawler access and structured data are not handled yet, fix those first, and a free AI visibility audit shows which gaps actually matter.
What is llms-full.txt and how is it different from llms.txt?
The difference is size: llms.txt is a short index of links, while llms-full.txt holds the full text of your key pages in one large markdown file.
The point of the full version is to let an AI read a site's substance in a single fetch, with no crawling at all. Documentation sites use it heavily, and Anthropic serves both files for its docs. When we fetched them on 7 November 2026, Anthropic's llms-full.txt was about 50 MB and ours was about 1 MB. For most agent, brokerage and service sites llms.txt alone is enough, because listings change too often to keep a full-text mirror current by hand.
3,768 words in 13 sections, 7 questions answered. Our charts and graphics are free to reuse (CC BY 4.0). Credit CiteVantage with a link to this page. Photographs are credited to their photographers and are not covered.
