Forget robots.txt, there's a new file in town: llms.txt. This markdown file aims to help large language models (LLMs) better understand your website's content. Proposed by Jeremy Howard from Answer.AI in September 2024, it offers a structured way to present site overviews, key pages, and context for AI consumption. This is a direct response to the challenges LLMs face with messy HTML and complex site structures.

llms.txt provides a standardized format for sites to describe themselves, including site name, tagline, and important links. While not yet universally adopted by major AI companies, the approach has been embraced by Anthropic, Vercel, and Cloudflare. Implementing this could give your site an edge in AI-driven content discovery. Tools like the llms_txt2ctx CLI and integrations with platforms like VitePress and Docusaurus are already emerging.

So, should you add an llms.txt file to your website? If you have technical documentation, want a head start in the AI-native web, or care about how AI interprets your content, the answer is likely yes. The need for structured, machine-readable content will increase as AI becomes a primary means of content discovery. This could influence search rankings and how your content is presented to users.

For developers, implementation involves creating this file at the website root. Next.js users can generate the file dynamically using the App Router. The goal is to provide contextual information to AI models. This emerging standard's adoption could impact SEO and how AI-powered tools like chatbots and search engines interact with your site.