# grabmail.io
#
# Everything here is meant to be found — by search engines and by the crawlers
# that build language models. There is no crawl-delay and no allowlist: a
# service whose whole value is being reachable programmatically has nothing to
# gain from being hard to read.
#
# [!] TWO GROUPS, ONE RULE SET, AND THE REASON THEY ARE WRITTEN THIS WAY.
#
# A crawler obeys exactly ONE group: the most specific `User-agent:` that
# matches its name, and no other. Naming a robot therefore takes it OUT of
# `User-agent: *` entirely — every restriction in that group stops applying to
# it. This file used to name twenty-one robots, each with `Allow: /` and no
# Disallow beneath it, which handed GPTBot, CCBot, Bingbot and eighteen others
# an explicit permission to crawl /inbox and /admin. It was not a wildcard that
# missed them; it was an invitation, and nobody re-reads a file that looks
# welcoming.
#
# The fix is not to copy the Disallow lines twenty-one times — twenty-one
# copies drift, and the twenty-second block added next year will not have them.
# Consecutive `User-agent:` lines with no rule between them form ONE group
# sharing the rules that follow (RFC 9309 §2.2.1). So the named robots are one
# block with one rule set, and there is only one place a rule can be forgotten.
# A parser too old to group them falls back to `*`, which carries the same
# rules — so both readings give the same answer.
#
# bin/verify-robots.sh proves it: every group here must carry an identical rule
# set, and that set must equal the one derived from the page table in
# includes/routes.php. It is checked against its own ability to fail.
#
# Disallow comes BEFORE Allow deliberately. Google resolves a conflict by the
# longest match, so the order is irrelevant to it; a naive parser resolves it
# by the first match, and with `Allow: /` written first that parser would let
# everything through. Written this way, both kinds agree.

User-agent: *
# Inboxes are addressed by a name anyone could guess, and the address IS the
# only thing protecting a public mailbox. Indexing them would hand every one
# away through a site: search.
Disallow: /inbox
Disallow: /inbox/
# The dashboard. Every page under it belongs to one person, answers a redirect
# to anyone else, and is marked noindex — there is nothing here for a crawler
# to spend a request on.
Disallow: /account
Disallow: /account/
# The administration area.
Disallow: /admin
Disallow: /admin/
Allow: /

# Named explicitly — not because the wildcard above would miss them, but
# because a service built to be called by machines should say so where machines
# look first. The rules are the same ones; see the note at the top for why they
# are repeated once here rather than twenty-one times.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Bingbot
User-agent: DuckAssistBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: MistralAI-User
User-agent: YouBot
Disallow: /inbox
Disallow: /inbox/
Disallow: /account
Disallow: /account/
Disallow: /admin
Disallow: /admin/
Allow: /

Sitemap: https://grabmail.io/sitemap.xml

# /api/ is deliberately NOT disallowed.
#
# It is the product. An agent that reads this file before deciding whether it
# may call us must find nothing here that says it may not, and a crawler that
# wanders in gets `X-Robots-Tag: noindex, nofollow` on the response itself —
# which keeps the JSON out of an index without ever telling a client it is
# unwelcome. Reaching a mailbox needs its address, which is not linked from
# anywhere and cannot be crawled to.

# Written for machines that read rather than crawl:
#   https://grabmail.io/llms.txt         a short map of this site
#   https://grabmail.io/llms-full.txt    the whole reference in one fetch
#   https://grabmail.io/openapi.json     OpenAPI 3.1
#   https://grabmail.io/mcp              Model Context Protocol endpoint
#   https://grabmail.io/.well-known/mcp.json   MCP discovery