API Documentation

The llms.txt Generator exposes two public JSON endpoints. POST /api/scan is protected by Turnstile (Cloudflare challenge) on the public web form, or by a free self-serve API key for direct callers. No signup required to use the form; sign up for a free API key for higher rate limits.

Endpoints

POST /api/scan

Start a crawl or fetch a cached result for a domain.

Submits a domain for a whole-site crawl. Returns 200 if a fresh result is cached (within 7 days), or 202 if the crawl is starting in the background — crawls can take a while (up to 200 pages), so poll GET /api/report/:domain.json for the result. Requires either a valid Turnstile token (from the web form), a free API key, or an internal authentication token.

POST /api/scan Request Body

{
  "domain": "example.com",
  "turnstileToken": "...",  // if from web form
  "forceRescan": false        // optional
}

Responses

Status	Meaning
`200`	Cache hit — result already fresh (within 7 days). Body contains the full crawl record.
`202`	Crawl started in background — poll GET /api/report/:domain.json until status is set.
`400`	Bad or unresolvable domain.
`429`	Rate limit exceeded. Wait retryAfterSeconds before retrying.

curl Example

curl -X POST https://llmstxt.unomage.com/api/scan \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"domain":"example.com"}'

Replace YOUR_TOKEN with your self-serve API key (starts with ultk_) from your account, or omit the Authorization header and include a valid turnstileToken if calling from a browser context.

GET /api/report/:domain.json

Fetch the full result for a domain.

Returns the complete scan record for a domain, including llmsTxt, llmsFullTxt, and diff against any existing file. If the domain has never been scanned, returns 404.

curl https://llmstxt.unomage.com/api/report/example.com.json

Rate Limits

3 fresh crawls per IP per hour (anonymous / Turnstile) — deliberately lower than a lighter-weight tool's, because a whole-site crawl is heavy.
3 fresh crawls per target domain per hour
20 fresh crawls per hour per self-serve API key

Cached results (within 7 days per domain) are served instantly and do not count against any of these limits.

Authentication

Three methods to authenticate a request to POST /api/scan: (1) Turnstile token from the web form (browser only), (2) free self-serve API key (Bearer ultk_...), or (3) internal authentication token (Bearer ...). GET /api/report/:domain.json needs no authentication.

Self-serve API key (free)

Don't have an API key? Sign up for a free account at /account/signup to generate your own with a higher rate limit.

Authorization: Bearer ultk_your_key_here