Blog

llms.txt is a proposed open standard for helping large language models (LLMs) better understand, navigate, and cite content from websites.

Placed at the root of a website at /llms.txt, the file acts as a curated, AI-friendly guide to a site’s most important content—written in plain Markdown so both machines and humans can read it easily.

Think of it as the modern evolution of robots.txt, but instead of telling crawlers what to avoid, it tells AI systems where to find your best content.

Historical Origins

The llms.txt standard was first proposed on September 3, 2024 by Jeremy Howard, Australian technologist, co-founder of fast.ai, and founder of Answer.AI.

Howard originally published the proposal as a blog post at answer.ai, framing it not as a finalized standard but as a starting point for community discussion and experimentation.

His motivation was practical: as LLMs increasingly rely on website content to assist users in real time—during inference, not just during training—the process of assembling relevant context from a site was ambiguous and inefficient.

Do you crawl the entire sitemap? Include external links? Include source code?

Howard’s answer was simple: let the site author decide, and give them a standardized file format to communicate that decision.

The Problem llms.txt Solves

Large language models face a fundamental technical constraint: context windows are too small to handle most websites in their entirety. Converting complex HTML pages filled with navigation menus, advertisements, JavaScript, and boilerplate into clean, usable text is both difficult and imprecise.

The result is that AI tools often pull from whatever they can parse fastest—which may include outdated pages, duplicate content, or low-signal sources rather than your most authoritative, carefully crafted content.

Without a guiding file, LLMs are essentially navigating your website blindfolded, guessing at what matters most.

llms.txt solves this by giving site owners a standardized way to say:

“Here are the pages that matter. Here’s what my site is about. Here’s how to understand it.”

Modern Definition

At its core, llms.txt is a plain text file written in Markdown, placed at the root directory of a website (yourdomain.com/llms.txt).

It provides:

• A concise description of the website or project
• Curated links to the most important and LLM-readable pages
• Optional contextual notes explaining what each linked resource covers
• Guidance on how to interpret the site’s content and structure

The file is specifically designed for inference-time use—meaning it helps LLMs give better answers to users right now, rather than being primarily aimed at training future models.

How llms.txt Differs from robots.txt and sitemap.xml

Many people initially compare llms.txt to existing web standards, but the differences are significant.

File	Purpose	Format	Audience	Content
robots.txt	Control crawler access	Custom syntax	Search engine bots	Allow/disallow rules
sitemap.xml	List all URLs for indexing	XML	Search engine bots	Full URL list with metadata
llms.txt	Curate key content for LLMs	Markdown	LLMs and AI agents	Structured links with descriptions

Unlike robots.txt, llms.txt contains no blocking or disallow directives—it is purely a positive, affirmative guide to your best content.

Unlike sitemap.xml, which lists every indexable page, llms.txt is a curated subset of the most important content, designed to fit within an LLM’s context window.

The File Format Explained

The llms.txt specification uses Markdown because it is the format most widely and easily understood by language models, while also remaining human-readable and parseable by standard programming tools.

Required and Optional Sections

A valid llms.txt file follows a specific structure in this order:

1. An H1 heading — The name of the project or site (only required element)
2. A blockquote — A short summary containing key information necessary for understanding the file
3. Body sections — Paragraphs or lists with more detailed background information
4. H2-delimited file lists — Sections containing curated URLs with optional descriptions
5. An “Optional” section — Links that can be skipped if a shorter context is needed

Example llms.txt File

Here is a simplified example based on the official specification:

# My Company Name

> We build project management tools for remote teams.

Our platform integrates with Slack, Notion, and GitHub.

## Docs

- [Getting Started](https://example.com/docs/start.md): Setup guide for new users
- [API Reference](https://example.com/docs/api.md): Full API documentation

## About

- [Company Overview](https://example.com/about.md): Mission and team

## Optional

- [Case Studies](https://example.com/case-studies.md): Customer success stories

The .md Companion Convention

The llms.txt proposal also includes a companion convention: making clean Markdown versions of web pages available at the same URL as the original page, but with .md appended.

For example, yoursite.com/blog/post would have a corresponding yoursite.com/blog/post.md that strips away HTML, navigation, and other non-content elements to deliver clean, LLM-digestible text.

llms-full.txt

A common variation is llms-full.txt, which expands the llms.txt index into a single large file containing the complete flattened text of the entire website rather than just links and descriptions.

This approach trades compactness for completeness, and some sites use both files simultaneously—the standard llms.txt for quick context assembly and the full version for deep site analysis.

Relationship to Generative Engine Optimization (GEO)

As AI-powered search engines like Perplexity, ChatGPT Search, and Google’s AI Overviews become primary discovery channels, a new discipline called Generative Engine Optimization (GEO) has emerged alongside traditional SEO.

While SEO focuses on ranking in search results, GEO focuses on being the source that AI systems cite when generating answers. llms.txt directly supports GEO strategy by:

• Clarifying canonical sources: Specifying which URLs represent your definitive content, reducing the risk of AI citing outdated or duplicate pages
• Prioritizing high-quality content: Ensuring thought leadership, case studies, and cornerstone content are visible to AI systems
• Protecting brand narrative: Directing LLMs to preferred content to reduce inaccurate or generic AI-generated descriptions of your business
• Supporting AI search indexes: Helping your content surface in Perplexity, ChatGPT, and other AI-driven search layers

Who Is Using llms.txt?

Adoption has grown rapidly since the proposal’s release in September 2024, though it remains concentrated in the developer and technology community.

Companies like Perplexity, Anthropic, and various developer documentation platforms have created llms.txt files for their own documentation and internal use.

As of June 2025, a scan of the top 1,000 most visited global websites showed approximately 0.3% adoption (3 out of 1,000 sites), suggesting the standard is still in early-adopter territory among mainstream web properties.

However, adoption among developer tools, SaaS documentation, and AI-adjacent companies is significantly higher.

Notable early adopters include:

• Anthropic: Uses llms.txt in internal documentation for agent-building
• FastHTML and nbdev projects: All fast.ai and Answer.AI software projects using nbdev automatically generate .md versions of all pages
• Perplexity: Has developed llms.txt files for its own documentation
• Mintlify, GitBook, and other documentation platforms: Have built native llms.txt generation into their platforms

Current Debate and Honest Limitations

It is important to approach llms.txt with measured expectations.

The standard remains a proposal, not an adopted protocol, and there are legitimate questions about its real-world impact.

Key Criticisms

LLMs may not actually read it: Critics point out that there is limited verified evidence that major AI systems—including ChatGPT, Claude, and Gemini—actively read and prioritize llms.txt files during inference. Google’s John Mueller has stated he doesn’t know of any search systems that use the file.

No enforcement mechanism: Like robots.txt, llms.txt can be obeyed or ignored by any AI agent—there is no technical enforcement. Its effectiveness depends entirely on voluntary adoption by LLM providers.

Not a ranking signal: llms.txt does not guarantee that a brand will appear in AI-generated answers. AI search does not operate based on a single file.

The illusory truth effect: The standard has spread rapidly through SEO and marketing communities, which some argue has outpaced actual evidence of its effectiveness.

The Measured Case for Adopting It Anyway

Despite these limitations, there are practical reasons to implement llms.txt:

• Google crawls it: Google has been observed crawling llms.txt files weekly, and in December 2025 stated that adding one would not harm a website.
• Legitimate AI crawlers check for it: Server logs show GPTBot, ClaudeBot, and PerplexityBot do access the file.
• Future-proofing: As AI search evolves, having a well-structured llms.txt positions you ahead of adoption curves.
• Content audit value: The process of creating llms.txt forces a useful exercise in identifying your most important content.
• Low implementation cost: Creating a basic llms.txt file takes minutes and costs nothing.

How to Create an llms.txt File

Step 1: Audit Your Most Important Content

Identify the pages that best represent your brand, products, services, and expertise. Focus on quality over quantity—the goal is a curated guide, not an exhaustive sitemap.

Step 2: Write Clear, Neutral Descriptions

LLMs perform best with content that clearly defines terms, avoids emotional language, and does not rely on context-free marketing claims. Instead of “an innovative, groundbreaking platform,” use “an analytics platform for monitoring and analyzing user behavior.”

Step 3: Structure the File in Markdown

Create your H1 title, add a blockquote summary, and build your H2 sections with linked file lists. Keep descriptions concise and informative.

Step 4: Place It at Your Domain Root

Host the completed file at yourdomain.com/llms.txt where AI crawlers expect to find it.

Step 5: Optionally Create .md Page Versions

For maximum AI readability, create Markdown versions of your key pages by appending .md to their URLs.

Step 6: Test and Monitor

Use a tool like llms_txt2ctx to expand your file into a full LLM context file and test whether AI systems can accurately answer questions about your content. Monitor server logs for bot access from GPTBot, ClaudeBot, and PerplexityBot.

Tools and Plugins for llms.txt

A growing ecosystem of tools supports llms.txt creation and management:

• llms_txt2ctx: Official CLI and Python module for parsing llms.txt files and generating expanded LLM context
• vitepress-plugin-llms: VitePress plugin that auto-generates LLM-friendly documentation
• docusaurus-plugin-llms: Docusaurus plugin for LLM-friendly docs following the llms.txt standard
• llms-txt-php: A PHP library for reading and writing llms.txt files
• Drupal LLM Support: A Drupal Recipe providing full llms.txt support for Drupal 10.3+ sites
• GitBook: Native llms.txt generation built into the documentation platform
• VS Code PagePilot Extension: Automatically loads external context from llms.txt files for enhanced responses

Use Cases by Industry

Software and Developer Documentation

The most natural fit for llms.txt—developers often use AI assistants while coding and need accurate, up-to-date library documentation. llms.txt ensures AI coding tools reference current API documentation rather than outdated information.

E-commerce

llms.txt can outline product categories, return policies, shipping information, and FAQ content—ensuring AI assistants give accurate answers when customers ask questions about a store.

Professional Services and B2B

Agencies, consultancies, and SaaS companies can use llms.txt to ensure AI systems accurately describe their services, expertise, and differentiators.

Publishing and Media

Content publishers can curate their most authoritative editorial content, helping AI systems cite original reporting rather than aggregated or republished versions.

Personal and Portfolio Sites

Individuals can use llms.txt to help AI systems accurately answer questions about their background, work, and expertise.

Future Outlook

The llms.txt standard sits at the intersection of several major trends reshaping the web: the rise of AI-powered search, the shift from click-based to answer-based information retrieval, and the growing importance of structured content for machine comprehension.

As AI search adoption accelerates—with OpenAI reporting roughly 700 million weekly active users and Google’s Gemini reaching 400 million monthly active users—the strategic importance of being accurately represented in AI-generated answers will only increase. Whether llms.txt becomes the definitive standard for AI content curation or is superseded by a more formalized protocol, the underlying principle—giving site owners a voice in how AI understands their content—is likely here to stay.

llms.txt represents a practical, low-cost step that website owners can take today to improve how AI systems understand and represent their content. While it is not a magic bullet for AI search visibility and its adoption by major LLMs remains inconsistent, the combination of minimal implementation cost, growing crawler interest, and the strategic importance of AI content optimization makes it a worthwhile addition to any modern web content strategy.

For web developers, marketers, and content strategists navigating the shift from traditional SEO to Generative Engine Optimization, llms.txt is less about immediate, measurable impact and more about positioning—ensuring your most important content is clearly organized, accurately described, and ready for the AI-driven web that is already here.

Introduction

The Big Idea is one of the most powerful and sought-after concepts in copywriting and direct-response marketing. It represents the single, compelling, unifying concept that captures a reader’s imagination, strikes them on a deeply emotional level, and compels them to take action. A Big Idea is the heartbeat of any successful sales promotion—the surprising, simple, and powerful hook that elevates ordinary copy into control-beating, cash-generating marketing material.

Historical Origins

The concept of the Big Idea has roots in the golden age of advertising, popularized by legendary ad man David Ogilvy, who famously declared that it takes a “Big Idea” to attract consumers’ attention and persuade them to buy. In direct-response copywriting, the Big Idea evolved into an even more critical element, as A-list copywriters like Gary Bencivenga, Clayton Makepeace, and Michael Masterson recognized that without a compelling central concept, even the most polished prose would fail to move the needle.

Over decades of direct mail, magalogs, and now digital promotions, the Big Idea has remained the single most important factor distinguishing winning promotions from losing ones. Marketers and copywriters have consistently identified it as the element that separates average campaigns from legendary controls that run for years and generate millions in revenue.

Modern Definition and Core Characteristics

A Big Idea is the overarching concept that drives an entire sales promotion—the unifying theme that ties headline, lead, body copy, and offer together into a cohesive, persuasive whole. According to AWAI and top copywriters, a genuine Big Idea has three essential characteristics:

• Surprising: It offers something unexpected and unique. If everyone is already talking about it in leading news headlines, it isn’t a Big Idea.
• Simple: It’s easy to understand and doesn’t make the reader think hard. Readers immediately grasp that you’re onto something meaningful.
• Powerful: It strikes the reader on a deeply emotional level, capturing their imagination and compelling them to learn more.

In other words, a Big Idea is…well, BIG.

Why Big Ideas Matter in Copywriting

The Foundation of Control Copy

Big Ideas are the foundation of “control” promotions—the packages that outperform all others and become the standard against which every new challenger is measured. Without a Big Idea, copy tends to blur into a sea of features, benefits, and claims that fail to distinguish themselves in the reader’s mind.

Emotional Resonance

A Big Idea doesn’t just inform—it moves. It connects with the reader’s deepest desires, fears, hopes, and frustrations, creating an emotional urgency that drives action. This emotional power is what transforms casual readers into eager buyers.

Competitive Differentiation

In saturated markets, a Big Idea provides the unique angle that makes a promotion stand out. It’s the reason a reader chooses to engage with your letter instead of tossing it aside with dozens of competing messages.

The Role of Curiosity

Marketers who hire direct-response copywriters consistently cite curiosity as the most important trait in the writers they hire. It’s this curiosity that leads to the kind of Big Idea that can result in control copy and bring in cash—both for the client and for the copywriter.

Finding a Big Idea is as simple as tapping into your own curiosity, asking lots of questions, and following the trails of information you discover further than other people are willing to go. The willingness to dig deeper is what separates A-list copywriters from the rest of the pack.

Proven Methods for Finding Big Ideas

Method 1: The Chain of Connections

One way to recognize a Big Idea is to look for something that makes you want to share it. The moment you encounter it in research, you find yourself thinking about who you can tell. It excites you.

When you find information you’re excited to share, you could be onto a Big Idea for a sales promotion. But sometimes you only find something mildly interesting—don’t stop there. Follow the idea further, ask questions, and dig deeper. The chain of connections often leads to something truly worthy of the Big Idea label.

Real-World Example: The Ronald Reagan Cancer Package

An A-list copywriter was hired to create a package selling a cancer information report—and the client wanted ONE big idea, not a bunch of small ones.

The writer first discovered that oxygen is beneficial against cancer. Interesting, but not share-worthy. Instead of giving up, he asked:

• “Who has used oxygen as a cancer therapy?”
• “What therapies are other countries using that aren’t available in the U.S.?”

Those questions led him to uncover that Ronald Reagan underwent cancer treatment WHILE serving as President of the United States—and that Reagan went to Germany for treatment, turning his back on American therapies.

The Big Idea: People in the know go outside of the U.S. for cancer treatment—this has been kept quiet by the media—and you need to know about it.

The Winning Headline: President Ronald Reagan’s Secret Victory Over Cancer

Method 2: Piling Up Several Similar Angles

Sometimes you don’t find a single Big Idea about a product. What you do find—if you look—are lots of ideas of all shapes and sizes that, when combined, become something much bigger.

Real-World Example: The Curcumin Promotion

Curcumin (an extract of turmeric) is a hot topic in the health field. Most writers stop researching when they find ONE big benefit and try to build a headline around it.

For instance: curcumin helps suppress abnormal cell growth (cancer). That’s good.

But if you keep looking, you’ll find there are OVER 600 diseases that show benefits from taking curcumin.

By going the extra mile, you actually have a BIGGER idea than you started with.

Possible Headline: This simple $1-per-day pill helps CURE over 600 diseases.

Key Elements of a Successful Big Idea

Surprise Factor

A Big Idea must surprise the reader with information or an angle they haven’t encountered before. Common knowledge and widely-reported news don’t qualify—by definition, a Big Idea offers something fresh.

Simplicity

The best Big Ideas can be communicated in a single, clear sentence. If a reader has to work to understand it, it’s not simple enough.

Emotional Impact

A Big Idea must connect with core human emotions—hope, fear, greed, curiosity, vanity, or love. Without emotional resonance, even the cleverest angle will fall flat.

Relevance to the Prospect

The Big Idea must matter deeply to the target audience. It should speak to their specific desires, pains, or aspirations in a way that generic messaging cannot.

Alignment with the Product

The Big Idea must naturally lead readers to the product or offer. If there’s a disconnect, the entire promotion loses credibility.

Where Big Ideas Come From

Deep Research

Big Ideas rarely come from surface-level research. They emerge from reading widely, following footnotes, exploring obscure sources, and making connections others miss.

Current Events and Trends

While headlines themselves aren’t Big Ideas, trends and events can be the starting point for finding connections that become Big Ideas.

Customer Conversations

Talking to actual customers—hearing their frustrations, desires, and language—often reveals Big Ideas hiding in plain sight.

Product Immersion

Using the product, understanding its origin story, and interviewing the people behind it can uncover unique angles that become Big Ideas.

Swipe Files and Competitive Analysis

Studying winning promotions from the past reveals patterns and structures that can inspire new Big Ideas for different products or markets.

Common Mistakes to Avoid

Confusing Features with Big Ideas

A product feature is not a Big Idea. The Big Idea is the emotional, surprising story that makes the feature matter.

Stopping at the First Interesting Finding

Many writers settle for the first interesting fact they discover. A-list copywriters keep digging until they find something truly share-worthy.

Chasing Trends

Popular topics already dominating the news usually don’t make Big Ideas. By the time a trend is obvious, it’s too late.

Overcomplicating the Concept

If your Big Idea requires paragraphs to explain, it’s not simple enough. Distill it until it fits in a single, punchy statement.

Ignoring the Audience

A Big Idea that excites the copywriter but doesn’t resonate with the target prospect is a failed Big Idea.

Big Ideas Across Formats

Long-Form Sales Letters

Traditional direct mail and online sales letters depend entirely on a strong Big Idea to sustain reader attention through thousands of words of copy.

Magalogs and Bookalogs

These hybrid formats use Big Ideas to create editorial-style content that feels informative while driving sales.

Video Sales Letters (VSLs)

Video scripts require Big Ideas just as much as text—the opening hook must deliver a surprising, emotional promise.

Email Promotions

Even short emails benefit from Big Ideas, though they may be expressed more concisely than in long-form copy.

Landing Pages

Modern web copy uses Big Ideas to differentiate offers in crowded digital marketplaces.

Testing and Refining Your Big Idea

The Excitement Test

Does the idea make you want to tell someone about it right now? If not, keep digging.

The Simplicity Test

Can you explain the Big Idea in one sentence that a stranger would immediately understand?

The Emotion Test

Does the idea evoke a clear emotional response—curiosity, hope, fear, outrage, or excitement?

The Uniqueness Test

Is anyone else in your market already using this angle? If so, it’s not big enough.

The Headline Test

Can you craft a compelling headline directly from the Big Idea? If the headline writes itself, you’ve likely found a winner.

Conclusion

Finding a Big Idea that gets you excited and overwhelms your reader with benefits, possibilities, curiosity, or emotion isn’t always easy—but it’s never impossible. All you need to do is dig deeper into the research and look for connections that will capture the imagination.

When you find a genuine Big Idea, you’ll be onto a winner. And that will help your client, your client’s customer, and you. The Big Idea remains the single most valuable skill a copywriter can develop—the one element that separates ordinary copy from the legendary promotions that define careers and generate fortunes. Master the art of finding Big Ideas, and you’ll never struggle to write copy that sells.