Methodology

The llms.txt Generator crawls a domain in stages: discovering URLs, classifying each into a section, writing a one-line description, summarising the site, then emitting spec-correct files and diffing against anything already published.

The pipeline

1. Discover: sitemap.xml first (following a sitemap index one level deep), falling back to a link crawl (depth 2) from the homepage when no usable sitemap is found. Hard cap: 200 URLs.
2. Classify: each URL into Docs, Blog, Services/Products, About, Legal, or Other — by URL pattern first, then page title/heading keywords.
3. Describe: one line per page, from your own meta description, or the first sentence of the first content paragraph, or the H1 — in that order. An LLM rewrite only runs when all three are missing or unusable (including the common case of a boilerplate meta description repeated across many pages).
4. Summarise: one blockquote sentence for the whole site — a single LLM call, with a plain template sentence as the fallback if no LLM key is configured or the call fails.
5. Emit: llms.txt per the proposal's format (H1, optional summary blockquote, H2 section lists), and llms-full.txt — clearly labelled as a widely-used convention, not part of the spec.
6. Diff: fetches any existing llms.txt already published at your root and compares it line-by-line against the newly generated one.

No score

This is a generator, not an audit — there is nothing meaningful to grade about a machine-readable index file. If you want a 0–100 read on how citable your actual content is, that's a different tool.

Edge cases

• No sitemap found → falls back to a link crawl, and says so in the generated file's notes.
• More than 200 pages → keeps the pages linked most directly from the homepage (or earliest in the sitemap) and states the cap plainly.
• JS-rendered navigation → if the homepage's raw HTML has no discoverable links, the tool says so rather than silently returning an empty file.
• Existing llms.txt already published → diffed, never silently overwritten.
• Every meta description identical across the site (extremely common) → detected and treated as boilerplate; falls back to the first-paragraph extraction instead.