About Unomage-LlmsTxtBot

Unomage-LlmsTxtBot is the crawler behind the llms.txt Generator. When someone submits a domain, this bot discovers and fetches up to 200 of that site's pages — the way an AI assistant's own crawler would — in order to build an llms.txt index.

How it identifies itself

Every request Unomage-LlmsTxtBot makes sends a standard identifying User-Agent header:

Mozilla/5.0 (compatible; Unomage-LlmsTxtBot/1.0; +https://llmstxt.unomage.com/bot)

so any site owner or server log can identify traffic from this tool and, separately, decide whether to allow it.

It respects robots.txt

Unomage-LlmsTxtBot obeys the target site's robots.txt rules for its own user agent by default. If a site's robots.txt disallows Unomage-LlmsTxtBot, this tool will not crawl it.

What it does not do

• It does not submit forms, click links beyond following <a href> URLs, or take any action beyond fetching pages.
• It does not store credentials or attempt to bypass logins or paywalls.
• It crawls at most 200 pages per domain, to a link-crawl depth of 2 — it is not a general-purpose, unbounded crawler.
• Requests to any one target domain are rate-limited (at most 3 fresh crawls per hour), so it cannot be used to hammer a site.

Concerns or exclusion requests

If Unomage-LlmsTxtBot is causing problems for your site, or you'd like requests from this bot to stop, contact us below.

For concerns or to request exclusion, contact: TODO — add a real contact email/address here before this ships to production.