# ONE group, ONE rule set — every agent listed below shares the rules that follow.
#
# A named `User-agent:` group REPLACES the `*` group rather than extending it
# (RFC 9309 section 2.2.1): a crawler obeys only the most specific group matching
# its own name, and groups are never merged. Listing each AI crawler in its own
# group with `Allow: /` and no Disallow lines therefore granted all seven of them
# /checkout/, /billing/, /accounts/, /dashboard/, /chat/, /insights/ and
# /ops-command-centre/ — the exact paths the `*` group above them disallowed.
#
# Not cosmetic: `GET /checkout/<price_id>/` creates a real Stripe Checkout
# Session (kept deliberately for bookmarked links — see `stripe_checkout` in
# website/views.py), so a crawl and an abandoned buyer produce identical records
# and abandoned-checkout data becomes unreadable. Same harm as bug-167.
#
# Stacking every agent into a single group means the disallow list is written
# ONCE and cannot drift — adding a private path is one line here, not eight.
# Naming the AI crawlers explicitly is deliberate and stays: Ops Intel sells GEO,
# so welcoming them on public content is positioning, not an oversight.
#
# The list grew on 2026-08-19. GPTBot is OpenAI's TRAINING crawler; OAI-SearchBot
# is the one that fetches pages to answer a live question, and only the first was
# named — so the site was open to being learned from and silent to being cited,
# which is backwards for a business that sells being cited. Google-Extended is
# the same distinction for Gemini, Applebot-Extended for Apple Intelligence, and
# CCBot feeds Common Crawl, which is an input to most of the rest.
User-agent: *
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: cohere-ai
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: YouBot
User-agent: CCBot

Allow: /

Disallow: /dashboard/
Disallow: /accounts/
Disallow: /ops-command-centre/
Disallow: /billing/
Disallow: /chat/
Disallow: /insights/
Disallow: /checkout/

Sitemap: https://www.opsintel.io/sitemap.xml
# LLMs.txt: https://www.opsintel.io/llms.txt
# Agent instructions (when to use Ops Intel): https://www.opsintel.io/agents.md
# Any page on this site can be fetched as Markdown — send Accept: text/markdown,
# or append .md to the path. See https://www.opsintel.io/agents/