Preparing Your Website for AI Agents: llms.txt, Structured Data and Markdown Pages for Agents

In brief: The agentic e-commerce products announced this month by Vercel and Shopify (details) describe shopping agents discovering stores through structured data, sitemaps and llms.txt, and product pages returning visuals to humans and Markdown to agents. The same trend shows on documentation sites: the Next.js blog and the Model Context Protocol site offer a /llms.txt index and Next.js blog posts have .md (Markdown) versions. So what should a service site or e-commerce site do? Proven and low-risk: clean HTML structure, structured data (schema.org), an up-to-date sitemap, fast and crawlable pages, clear contact/company information. Worth trying but unproven: llms.txt and Markdown versions for agents. I can't offer a reliable measurement of llms.txt's effect on search ranking or AI answers; nor is it presented as an official web standard. This article describes current practices and their limits.

Last updated: 26 October 2026

How do agents “read” your site?

An AI agent or assistant can reach your site in two ways:

1. Fetch the page like a browser and extract the text. Then complex JavaScript, hidden content or slow pages cause trouble.
2. Take data from dedicated channels: MCP servers, protocols (such as UCP) or structured data your site offers.

In both the goal is the same: that the agent correctly understands what you offer, its price, its terms and how to contact you.

Proven, low-risk foundations

1. Clean, meaningful HTML

A heading hierarchy (h1, h2…), descriptive link text, tables and lists help both people and automated readers. Pages whose content loads only afterwards via JavaScript may be invisible to readers that don't extract text. That is why server-side rendering matters: our technical SEO with Next.js guide.

2. Structured data (schema.org)

Product, organisation, service, FAQ and article markup makes details like price, stock, address and opening hours machine-readable. In the Vercel/Shopify announcement structured data is also among the ways listed for agent discovery. Tools like Google's Rich Results Test can verify it.

3. An up-to-date sitemap and correct crawl settings

Keep sitemap.xml current and accurate; don't block important pages by accident with robots.txt. On multilingual sites hreflang and canonical links must be consistent: our multilingual site guide.

4. Speed and stability

Slow and error-prone pages give a bad result for both users and automated readers.

5. Clear, verifiable information

Your services, price ranges (if any), processes, contact details, FAQs and policies should be written plainly. Agents can quote information from your page when answering users; vague information gets quoted wrongly. For visibility in AI search, see our GEO guide.

Worth trying but unproven: llms.txt

llms.txt is a text (Markdown) file at the site root that lists the site's important pages, with short descriptions, in a readable form for AI models/agents. Examples seen in practice:

• The Next.js blog: “For an index of all Next.js content, see /llms.txt.”
• The Model Context Protocol site: “Fetch the complete documentation index at /llms.txt.”

Why caution?

• It is not an official web standard; it is a practice proposed by the community.
• I have no verifiable evidence here that search engines or AI assistants use it for ranking or quoting.
• It makes more sense for documentation sites; on a small service site the gain may be limited.

When is it worth trying? If you have many documentation/content pages, a product documentation set or a broad knowledge base. It is a low-cost file; trying it is usually harmless, but keep expectations low. Remember the maintenance load too: if the file isn't kept up to date it misleads.

A simple llms.txt example (Markdown):

# Company Name

> One or two sentences on what the company does.

## Services
- [Web applications](address): A short description of what is offered
- [E-commerce solutions](address): A short description

## Resources
- [Blog](address): Guide articles
- [Contact](address): Quotes and support

Serving Markdown to agents

Vercel Shop returns product pages to agents as Markdown (description, price, options, stock) while showing humans the visual page. The Next.js blog also has .md Markdown versions of its posts. The advantage: the agent gets clean content without layout noise.

Things to watch:

• Consistency between the two versions is essential: don't let the Markdown version carry different or missing information.
• Show users and agents the same facts as content. Showing search engines different content (“cloaking”) can violate spam policies; so if you add an agent-oriented version, mind Google's rules and our spam update guide. Do it only to present equivalent content in a more machine-friendly form and with expert advice.
• This is an advanced practice; for most service sites it isn't the first priority.

Priority order for service sites

Extra steps for e-commerce sites

• Consistency and freshness in product data (price, stock, variants),
• Live stock integration,
• Clear texts for payment/return/shipping policies,
• Analytics that tell agent-sourced traffic and orders apart.

For details see our agentic e-commerce article.

Security and privacy notes

• Every channel you open to agents (API, MCP server, Markdown endpoint) is an attack surface: our MCP guide and agent access controls.
• Don't put secret or internal information in publicly accessible files (llms.txt included).
• Apply rate limiting and abuse protection to agent-sourced requests.

AI-ready websites with Aktaş Digital

We build modern, fast, structured-data-rich and agent-ready websites. To have your current site audited for technical SEO and agent readiness, use our quote form. Related services: web applications.

Frequently Asked Questions

Is llms.txt mandatory?

No. It is not an official standard, and there is no proven ranking effect. It can be tried on documentation-heavy sites.

Does Google read llms.txt?

I have no source to verify that. So I make no promise about ranking or visibility.

Should I make separate Markdown pages for agents?

For most service sites it isn't the first priority. If you do, make sure the content is consistent with the human version.

What are the three most important things?

Clean and fast HTML, correct structured data and clear, up-to-date content.

How do I appear in AI answers?

No guarantee can be given. Original, accurate, well-structured content and trust signals are the soundest approach: our GEO guide.

Conclusion

Preparing your site for agents is mostly a continuation of what a good website needs anyway: clean HTML, structured data, an up-to-date sitemap and clear content. llms.txt and Markdown versions are cheap experiments, not proven gains. First step: open your five most important pages with JavaScript turned off and check whether the text content is visible.