There is something quietly revolutionary brewing on the internet. It isn’t a flashy new app or some viral trend. It’s a plain text file with a simple name: llms.txt and in our opinion, it might just become one of the most impactful tools in how websites interact with artificial intelligence. As the web continues to be shaped and reshaped by large language models (LLMs), we believe this little file could help restore some balance between content creators and AI companies.
At first glance, llms.txt doesn’t sound like a big deal. It might remind you of the more familiar robots.txt file that websites use to tell search engines which pages to crawl and which to leave alone. But llms.txt is different. It isn’t about web crawlers or SEO in the traditional sense. Instead, it’s about how LLMs like ChatGPT, Claude, Gemini and others access and use the content that publishers and creators are putting out into the world.
In our view, the introduction of llms.txt is both timely and necessary. AI tools are becoming central to how people gather information, make decisions, and even learn. But the relationship between LLMs and the websites they pull data from has been murky at best. There’s very little transparency, even less control, and hardly any agreement on what is fair use. That’s where llms.txt comes in.
The Birth of llms.txt
The idea behind llms.txt was introduced by Jeremy Howard, the co-founder of Fast.ai and Answer.AI. Howard is known for advocating open access to technology and education, but he’s also very aware of the power dynamics between massive AI models and the people who create the data they are trained on.
We think what sparked the creation of llms.txt is the growing concern that LLMs are scraping content from across the web without giving credit or even acknowledging the source. This has become a major ethical and economic issue for publishers, newsrooms, and anyone who makes a living from original content. If an AI model is trained on your hard work and then turns around and summarises it without attribution or compensation, it feels like the rules of the web are being rewritten and not in your favour.
So what does llms.txt actually do? It’s a text file that sits in the root directory of a website and provides a structured outline of what content is available and how AI systems should interpret it. Think of it as a guidebook for LLMs, written by the content creators themselves. In our opinion, it’s an elegant solution to a messy problem.
How llms.txt Works in Practice
Unlike traditional formats that are optimised for search engines or visual layout, llms.txt is written in Markdown. It is designed to be simple, lightweight, and easy for both humans and machines to read. The structure typically includes a title, a short summary of the site or project, and a categorised list of links to key pages.
From a technical point of view, it’s brilliantly straightforward. But the impact could be far-reaching. By guiding AI systems to the content that matters most, llms.txt could make it easier for models to understand context, improve the quality of their summaries, and reduce the risk of misinformation. At the same time, it gives publishers a chance to take back some control over how their content is being used.
We believe this is important because the current model is, quite frankly, broken. AI companies scrape content from the open web with little to no transparency. They rarely disclose exactly which sources are used for training or inference. And they certainly don’t ask permission from the people who created the content in the first place. Llms.txt doesn’t solve all of these problems, but it does offer a practical step toward accountability.
Why It Matters for Publishers and the Future of the Web
In our view, one of the most exciting aspects of llms.txt is that it empowers publishers in a way that doesn’t rely on lawsuits or massive infrastructure. It’s easy to implement. You don’t need a dev team or a legal department to set it up. You just write a text file, upload it to your server, and you’re done.
That kind of simplicity is what the open web was built on. And in our opinion, it’s what makes llms.txt so compelling. It doesn’t try to fight AI with red tape. Instead, it invites collaboration by making it easier for AI systems to do the right thing.
It also has the potential to improve the experience for users. If LLMs have clearer access to structured, high-quality content, the information they provide will be more accurate and more useful. That means fewer hallucinations, better source attribution, and a more trustworthy AI experience overall.
We think it’s also worth noting that llms.txt doesn’t need to be confrontational. It’s not about blocking AI. It’s about guiding it. It tells LLMs: “Here’s what we want you to see. Here’s what’s important. And here’s where it came from.” That spirit of cooperation, rather than resistance, might be what makes this standard actually catch on.
Adoption Is Still in Early Days
As of now, the adoption of llms.txt is still very much in its infancy. Major LLM providers like OpenAI, Google, and Anthropic haven’t officially endorsed the format, nor have they confirmed that their systems are actively reading or honouring these files. But some developers and platforms are already embracing it.
Tools like Mintlify and Cursor, which are used for documentation and coding assistance, have begun generating llms.txt files automatically. We believe that as more websites and CMS platforms add native support for this format, it will become harder for AI companies to ignore.
That said, there’s still a long way to go. For llms.txt to work as intended, it needs to become a widely recognised and respected standard. That means AI companies need to agree on how to read the files, what weight to give the information inside them, and how to balance those signals with other data sources. It also means publishers need to get on board and start adopting it across industries and use cases.
In our opinion, the key will be transparency. If AI companies start publishing documentation that explains how they interact with llms.txt, that could go a long way toward building trust. And if publishers see tangible benefits, such as better traffic attribution or improved representation in AI responses, adoption will likely follow.
What We Hope to See Next
We believe that llms.txt could be a critical building block in a new kind of relationship between the web and AI. It’s not a silver bullet, and it certainly won’t fix everything. But it’s a start. And sometimes a small, simple tool is all it takes to shift the balance.
What we hope to see next is wider awareness. More education. More discussion about how AI and content creators can coexist in a way that is fair, transparent, and mutually beneficial. We also hope that major AI players begin to acknowledge the importance of giving creators a voice in how their content is used.
Because at the end of the day, the web is made by people. Not algorithms. Not models. But writers, coders, artists, educators, and dreamers. If we’re building AI systems that rely on that content, we owe it to those people to respect their work and give them a say in how it’s used.
Final Thoughts
In our view, llms.txt represents something more than just a new file format. It’s a symbol of the kind of internet we want to build, one where humans still have agency, where creators are respected, and where technology works in partnership with people, not in place of them.
The early days of the web were defined by openness and cooperation. Over time, things became more centralised and more commercial. But with tools like llms.txt, we think there’s a chance to return to something a bit more human, a bit more fair, and a bit more transparent.
If you run a website, we encourage you to explore llms.txt for yourself. If you’re building AI tools, we hope you consider supporting it. And if you’re simply someone who believes that creators deserve a voice in the future of the web, then you’re already part of the conversation.
Let’s keep building that better web, one simple text file at a time.
Contact us today to learn more about your Digital Marketing Strategies.
By Manesh Ram, Digital Marketing Specialist. Please follow @maneshram & Meta
