robots.txt and llms.txt: control what crawlers and models are shown

Every site writes robots.txt (for crawlers), llms.txt, llms-full.txt and llms-guides.txt (for models) from what its spaces publish. You shape them with two settings on the site: robots and llms. Edit them in the console under the site's Crawlers and AI files (with a live preview), over the API at GET/PUT/PATCH /orgs/{orgId}/sites/{siteId}/robots and …/llms (both take ?dryRun=true), or in forumulate.yaml under sites[].config. Changes reach the files within seconds, on the platform address, on custom domains and on site mounts.

robots

Merge order, from the strongest: the generated rules for deindexed and private spaces are always kept, in every group. Then a crawler's own AI policy, then the global AI default, then extraRules, disallowPaths, sitemaps, and last your raw block.

The AI crawlers: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, DuckAssistBot, YouBot. If every space is closed to indexing the file is a flat refusal and nothing here can open it.

sites:
  - slug: main
    config:
      robots:
        aiCrawlers: disallow
        aiCrawlerPolicy: { ClaudeBot: allow, GPTBot: allow }
        disallowPaths: ["/drafts/"]
        extraRules:
          - { userAgent: bingbot, crawlDelay: 5 }
        extraSitemaps: ["https://example.com/news.xml"]

Previewing a robots.txt before saving it

POST /orgs/{orgId}/sites/{siteId}/robots/preview takes a draft robots object (the same shape as the PUT body) and answers { "text": "…" }, which is exactly the file the site would serve, merged with the generated rules. Nothing is stored. The console's live preview uses it.

curl -X POST "https://forumulate.com/api/orgs/$ORG/sites/$SITE/robots/preview" \
  -H "Authorization: Bearer $FORUMULATE_TOKEN" -H "Forumulate-Version: 2027-03-01" -H "Content-Type: application/json" \
  -d '{"aiCrawlers":"disallow","aiCrawlerPolicy":{"ClaudeBot":"allow"}}'

llms

A single page opts out with llms: false in its metadata or front matter. The guides split into as many llms-guides-<n>.txt parts as needed (each at most 2 MB), llms.txt links every part, and a part past the last is a 404.

Where each file is served

The site files sit at the root of the host the site owns: https://docs.example.com/llms-guides.txt when the site is the whole domain, or /s/{org}/{site}/llms-guides.txt on our address. When your own app serves the host root and forwards only the space paths (/docs, /guides) to us, the root files are your app's to forward or serve. A space's files always answer under its path, so https://example.com/docs/llms-full.txt?part=guides is the guides-only file for that space.

sites:
  - slug: main
    config:
      llms:
        title: Acme for models
        summary: Docs and API reference for Acme.
        exclude: { pages: ["internal-*"] }
        sections: [{ space: docs, title: Documentation }]
        full: { maxBytes: 4000000 }

llms-quickstart.txt: the short file

llms.txt indexes every guide, which is the right file for a crawler and too long for an agent that wants to answer one question. /llms-quickstart.txt, beside it on the site's address, is the page to read first: your title, summary and intro, the first six pages of each space (your sidebar order is the order of importance, and llms: false pages and include/exclude are already applied), and the two ways to go further: the site search (GET /public/orgs/{orgId}/sites/{slug}/search?q=, anonymous, JSON hits with absolute URLs; see Site search) and the full llms.txt. It goes with the rest when enabled: false.

Files are served with Cache-Control and a strong ETag (a matching If-None-Match answers 304), and are refreshed when a page or the site changes.

IndexNow: telling search engines a page changed

robots.txt and the sitemap wait for a crawler to come back; IndexNow tells Bing, Yandex, Seznam and Naver at once. Switch it on per space — studio → Settings → Search engines (IndexNow), or PUT /orgs/{orgId}/spaces/{spaceId}/publish-pings with {"indexNow": true} (PATCH works too), or settings: { publishPings: { indexNow: true } } on the space in forumulate.yaml (any space type, the same key). Off until you switch it on.

Forum, status, contact, events, roadmap and content spaces do not ping: indexNow: true on one answers 422 validation.failed with details.indexNow, rather than storing a switch that never fires.

Only what the sitemap would list is pinged. Nothing goes out for a private space, one behind reader sign-in, or one set to noindex, nor for a page set to noindex or limited to an audience. A page edited in quick succession is pinged at most once every ten minutes; a rename or a delete always goes out.

The key file is served at <space root>/indexnow.txt while the switch is on, and answers 404 otherwise. That is /p/{org}/{space}/indexnow.txt on our address, https://docs.yourcompany.com/indexnow.txt on a domain of its own, and https://yourcompany.com/docs/indexnow.txt under a path mount (the IndexNow spec accepts a key in a folder for the addresses under that folder). The API answers the same key at GET /public/orgs/{orgId}/spaces/{spaceId}/indexnow-key and …/indexnow.txt. The key is derived from the space and never changes, and it is the one pings are signed with, so you cannot supply your own.

In the API

Every operation below is also reachable from the other surfaces without a dedicated command. forumulate api <Operation> calls it from the CLI, the JavaScript SDK has a typed method for each one (the last column names it; the Python and Go SDKs are generated from the same spec), and an agent reaches the organization-scoped ones through the MCP agent server — its api_* family tools, or call_api. Where a CLI command or a hand-made MCP tool exists too, the last column names it. Operation names link to the API reference.

Public Site

Site Machine Files

Site Robots Preview