Test an email verification flow end to end

Why the mail half is usually untested

Almost every sign-up test stops at the same place: fill in the form, assert that the page says “check your inbox”, done. Everything after that sentence — whether the mail was actually sent, whether the code in it works, whether the confirmation link goes anywhere — is left to production.

It is left there because reading mail from a test is awkward, and the usual ways round it each prove something slightly different from what you wanted:

Approach	What it proves	What it misses
Mock the mailer	Your code called `send()` with the right arguments.	Everything after `send()`: the template, the link, the DNS, the fact that the provider rejected the message.
A local catch-all SMTP (MailHog, Mailpit, smtp4dev)	The message left your application and was well formed.	Anything that only happens with a real MX lookup, a real provider, or a real recipient domain. And it is one more service to run in CI.
A real disposable mailbox	The message left your application, crossed the public internet, was accepted by a real mail server, and the code inside it works.	Nothing much — but it is asynchronous, so the test has to be written to wait properly.

This guide is about the third one. The awkward part is not the API — it is three calls with no key — but the waiting, and that is where flaky tests come from.

The shape of the loop

Three steps forward and one poll back. The only edge worth arguing about is the dashed one: it is a poll with a deadline, not a sleep.

There are three endpoints and you will use two of them:

GET /api/v1/mailbox: Everything waiting at an address, newest first. This is the call the loop polls. An empty mailbox answers 200 with count: 0.
GET /api/v1/message/{id}: The full message: headers, the plain-text part, the HTML part, attachments. html is null when the sender sent text only.
DELETE /api/v1/message/{id}: Optional. Everything expires after 5 days anyway, but deleting keeps a shared mailbox clean between runs. Idempotent.

No API key, no account and no headers on the public domains. If your CI needs a secret to run these tests, something has gone wrong somewhere else.

A fresh address for every run

This is the single most important line in the whole setup, and it is one line. Nothing has to be created, so a new address costs nothing: derive it from the build id, or from eight random characters, and never reuse one.

shell

$ ADDR="ci-$(openssl rand -hex 4)@grabmail.io"; echo "$ADDR"

If your CI runs jobs in parallel, this also removes the whole class of problems where two jobs read each other’s mail. Different address, different mailbox, no coordination needed.

Prefix them with something recognisable — ci-, e2e-, the branch name — so that when you are looking at a mailbox by hand at two in the morning you know which pipeline made it.

Polling, with a deadline

Mail is not synchronous. It normally lands in a couple of seconds and occasionally takes twenty, so the test has to wait — and how it waits decides whether the suite is trustworthy.

A deadline, not a retry count. for i in 1..30 means “thirty attempts at whatever speed the loop happens to run”, which silently becomes a shorter timeout as the API gets faster and a longer one as your sender gets slower. A wall-clock deadline means the same thing on every machine.
One request per second. That is the intended rhythm and it is never throttled. Faster is refused with 429, and it would not help anyway.
No special case for empty. An empty mailbox is 200 with count: 0, so the loop only ever looks at messages[0].
Fail loudly at the deadline. “No message within 60s” is a useful failure. A test that hangs until the CI job is killed is not.

poll until it arrives, or give up at 60 seconds

ADDR="ci-$(openssl rand -hex 4)@grabmail.io"
DEADLINE=$(( $(date +%s) + 60 ))
ID=""

while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  ID=$(curl -sG https://grabmail.io/api/v1/mailbox \
         --data-urlencode "address=$ADDR" \
       | jq -r '.messages[0].id // empty')
  [ -n "$ID" ] && break
  sleep 1
done

[ -n "$ID" ] || { echo "no mail for $ADDR after 60s" >&2; exit 1; }

The status codes you will actually meet

Code	Means	What the loop should do
`200`	The mailbox was read. `count` may be `0`.	Look at `messages`; if empty, wait a second and go again.
`400`	The address is missing or malformed.	Fail immediately. Retrying will not fix a typo.
`404`	That domain is not hosted here.	Fail immediately, and check the MX record if it is your own domain.
`429`	More than one read per second for that address, or more than 1200 requests a minute from this client.	Sleep for the number of seconds in `Retry-After`, then continue. Do not fail the test.

Getting the code, or the link, out of the message

Read the message by its id and you get both parts of it. Which one you should be parsing depends on what your application sends:

text: The plain-text part. Parse this when there is one — it is stable, it has no markup in it, and a six-digit code is a six-digit code.
html: The HTML part, or null if the sender did not include one. Confirmation links often only exist here, wrapped in an <a href>.

pull a six-digit code out of the plain-text part

curl -sG "https://grabmail.io/api/v1/message/$ID" \
  --data-urlencode "mailbox=$ADDR" \
| jq -r '.text' | grep -oE '[0-9]{6}' | head -1

or a confirmation link out of whichever part has one

curl -sG "https://grabmail.io/api/v1/message/$ID" \
  --data-urlencode "mailbox=$ADDR" \
| jq -r '[.text, .html] | map(select(. != null)) | join(" ")' \
| grep -oE 'https://[^"<> ]+/confirm/[A-Za-z0-9._-]+' | head -1

In Playwright

One helper module, used by every test that needs mail. It handles the deadline, the 429 and the “which message” question in one place, so the tests themselves stay readable.

tests/helpers/mailbox.ts

const API = 'https://grabmail.io/api/v1';
const DOMAIN = 'grabmail.io';

export type Summary = { id: string; from: string; subject: string; date: string };
export type Message = { id: string; subject: string; text: string | null; html: string | null };

const sleep = (ms: number) => new Promise(r => setTimeout(r, ms));

/** A mailbox nothing else in this run, or any previous run, is using. */
export function freshAddress(prefix = 'ci'): string {
  return `${prefix}-${Math.random().toString(36).slice(2, 10)}@${DOMAIN}`;
}

/**
 * Block until a message arrives, or the deadline passes.
 * Polls once a second, which is the documented rhythm and is never throttled.
 */
export async function waitForMessage(
  address: string,
  opts: { timeoutMs?: number; subjectContains?: string } = {},
): Promise<Summary> {
  const deadline = Date.now() + (opts.timeoutMs ?? 60_000);

  while (Date.now() < deadline) {
    const res = await fetch(`${API}/mailbox?address=${encodeURIComponent(address)}`);

    if (res.status === 429) {                        // slow down, do not fail
      await sleep(Number(res.headers.get('retry-after') ?? 1) * 1000);
      continue;
    }
    if (!res.ok) throw new Error(`GET /mailbox answered ${res.status}`);

    const { messages } = (await res.json()) as { messages: Summary[] };
    const hit = opts.subjectContains
      ? messages.find(m => m.subject.toLowerCase().includes(opts.subjectContains!.toLowerCase()))
      : messages[0];
    if (hit) return hit;

    await sleep(1000);
  }
  throw new Error(`no message for ${address} within the deadline`);
}

export async function readMessage(address: string, id: string): Promise<Message> {
  const res = await fetch(`${API}/message/${id}?mailbox=${encodeURIComponent(address)}`);
  if (!res.ok) throw new Error(`GET /message answered ${res.status}`);
  return res.json() as Promise<Message>;
}

/** Anchored on your own wording, so a template rewrite fails the test loudly. */
export function codeFrom(message: Message): string {
  const body = `${message.text ?? ''} ${message.html ?? ''}`;
  const m = body.match(/code is\s*([0-9]{6})/i);
  if (!m) throw new Error('no confirmation code in the message');
  return m[1];
}

The test that uses it reads like the feature it is testing, which is the whole point of putting the waiting somewhere else:

tests/signup.spec.ts

import { test, expect } from '@playwright/test';
import { freshAddress, waitForMessage, readMessage, codeFrom } from './helpers/mailbox';

test('a new account can confirm its address', async ({ page }) => {
  const address = freshAddress();

  await page.goto('/signup');
  await page.getByLabel('Email').fill(address);
  await page.getByRole('button', { name: 'Create account' }).click();
  await expect(page.getByText('Check your inbox')).toBeVisible();

  const summary = await waitForMessage(address, { subjectContains: 'confirm' });
  const message = await readMessage(address, summary.id);

  await page.getByLabel('Confirmation code').fill(codeFrom(message));
  await page.getByRole('button', { name: 'Confirm' }).click();

  await expect(page.getByRole('heading', { name: 'Welcome' })).toBeVisible();
});

Set timeout on the test to comfortably more than the mail deadline — a 60-second wait inside a 30-second test is a test that always fails at 30 seconds and tells you nothing about the mail.

In pytest

The same three pieces: a fresh address, a wait with a deadline, and an extractor anchored on your own template.

tests/mailbox.py

import re
import secrets
import time

import requests

API = "https://grabmail.io/api/v1"
DOMAIN = "grabmail.io"

CODE = re.compile(r"code is\s*(\d{6})", re.I)


def fresh_address(prefix: str = "ci") -> str:
    """A mailbox no other run is using. Nothing has to be created first."""
    return f"{prefix}-{secrets.token_hex(4)}@{DOMAIN}"


def wait_for_message(address: str, timeout: float = 60, subject_contains: str | None = None) -> dict:
    deadline = time.monotonic() + timeout

    while time.monotonic() < deadline:
        r = requests.get(f"{API}/mailbox", params={"address": address}, timeout=10)

        if r.status_code == 429:                      # slow down, do not fail
            time.sleep(float(r.headers.get("Retry-After", 1)))
            continue
        r.raise_for_status()

        for m in r.json()["messages"]:
            if subject_contains is None or subject_contains.lower() in m["subject"].lower():
                return m
        time.sleep(1)

    raise AssertionError(f"no message for {address} within {timeout}s")


def read_message(address: str, message_id: str) -> dict:
    r = requests.get(f"{API}/message/{message_id}", params={"mailbox": address}, timeout=10)
    r.raise_for_status()
    return r.json()


def code_from(message: dict) -> str:
    body = f"{message.get('text') or ''} {message.get('html') or ''}"
    m = CODE.search(body)
    assert m, "no confirmation code in the message"
    return m.group(1)

tests/conftest.py

import pytest

from .mailbox import fresh_address


@pytest.fixture
def inbox() -> str:
    """A brand-new mailbox for this test, and this test only."""
    return fresh_address()

tests/test_signup.py

from .mailbox import code_from, read_message, wait_for_message


def test_signup_confirmation(client, inbox):
    client.post("/signup", data={"email": inbox, "password": "hunter2hunter2"})

    summary = wait_for_message(inbox, subject_contains="confirm")
    message = read_message(inbox, summary["id"])

    response = client.post("/confirm", data={"email": inbox, "code": code_from(message)})
    assert response.status_code == 200

Making it survive CI

Everything above works on a laptop. These are the things that only break once it is running twenty times a day on somebody else’s machine.

Symptom	Cause	Fix
Passes locally, fails in CI	The CI runner cannot reach the public internet, or egress is filtered.	Allow `grabmail.io` over HTTPS. There is nothing else to allow — no SMTP, no inbound.
Passes on rerun, fails on first try	Your sender queues mail and the deadline is shorter than the queue.	Raise the deadline before you touch anything else. Sixty seconds is a reasonable ceiling for a transactional mail.
`429` in bursts	Several tests polling the same address, or more than 1200 requests a minute from the runner.	One address per test. The per-address limit is one read a second; the client ceiling is 1200 a minute, which is twenty mailboxes polled once a second.
Green build, broken feature	A reused address served an old message.	A fresh address per run. This is the one that actually matters.
Flaky only in parallel	Two tests sharing a mailbox, or an assertion on which message is newest.	Filter by subject rather than taking `messages[0]` blindly.
Works for a week, then never	You are asserting on a message older than 5 days, cached somewhere.	Nothing survives 5 days here. Fixtures must create their own mail.

Cleaning up

Optional, because everything expires anyway — but a run that deletes what it read starts the next one from a mailbox that is genuinely empty, which makes a failure much easier to read.

shell

$ curl -sX DELETE -G "https://grabmail.io/api/v1/message/$ID" \
  --data-urlencode "mailbox=$ADDR"

It is idempotent: deleting twice still answers 200, so a cleanup step never fails a build on its own.

If an agent is driving the browser

A language model working through a sign-up flow hits the same wall, with one difference: it cannot afford to poll, because every turn of the loop costs tokens.

There is an MCP server at https://grabmail.io/mcp for exactly this, with no key and no account. Six tools, and the one that matters here is wait_for_message: it blocks for up to twenty-five seconds and returns the message in full, or says it timed out so the agent can simply call again.

mcp.json

{"mcpServers":{"grabmail":{"url":"https://grabmail.io/mcp"}}}

It can also wait for a particular message — pass subject_contains or from_contains and anything else that arrives is ignored, which is the same discipline as filtering by subject above.

That is the short version, and it is enough for a test that happens to be driven by a model. Giving an agent an inbox of its own is the long one: connecting a client, the four calls end to end, why the wait comes back before the mail does, and what to put in the agent’s own instructions so it gets this right every time.

There is also an OpenAPI 3.1 spec for code generators, and llms.txt for anything that would rather read plain text than crawl a page.

Before you call it done

A different address every run, derived from randomness or the build id.
A wall-clock deadline, and a failure message that names the address it waited on.
429 handled by waiting out Retry-After, not by failing.
The code or link matched against your own wording, not a bare [0-9]{6}.
The test timeout comfortably larger than the mail deadline.
No assertion on how fast the mail arrived — only that it did.
Nothing in the suite depending on a message that is more than 5 days old.

That is the whole discipline. Everything else about testing mail is the same as testing anything else asynchronous.

Questions

Do I need an API key for this?

No. The public domains take no key, no account and no headers. A domain that has been given a key of its own sends Authorization: Bearer <key>; nothing else does.

How fast does mail actually arrive?

Usually within a couple of seconds of your application handing it to its provider. It is not synchronous, though, and no test should assert on the delay — only that the message arrives before a deadline you chose.

How many mailboxes can I poll at once?

The per-address limit is one read per second. The per-client ceiling is 1200 requests a minute, which is exactly twenty mailboxes polled once a second. Past either you get 429 with Retry-After.

Can I use my own domain for CI?

Yes, and it is the better answer if your application refuses known disposable domains at sign-up. One MX record, nothing to register — the setup is here.

Is a mailbox private while my test is using it?

No. Anyone who knows the address can read it, on a public domain and on your own. For CI that is usually irrelevant — a random address that exists for eleven seconds and holds one throwaway code — but do not point a staging environment that sends real customer mail at it.

What happens to the messages afterwards?

They are deleted 5 days after they arrive, whether the test read them or not. Deleting them yourself at the end of a run is optional and idempotent.