GitHub Actions: an end-to-end job that reads real email

What changes on a runner

The test itself does not change. What changes is everything around it, and each of these has a specific answer rather than a shrug:

There is no secret to mount

Reading a public mailbox needs no key, no account and no header, so the mailbox side adds nothing to secrets. The only credential in the job is the one your application already needs to send mail — SendGrid, Postmark, SES, whichever — and it belongs to your app, not to the test.

The runner has to reach the internet

Outbound HTTPS to grabmail.io, and outbound to whatever your mailer uses. GitHub-hosted runners allow both by default; a self-hosted runner behind an egress filter needs one rule added.

Runs overlap

Two pull requests, four shards, a retry of a flaky job — several copies of the same test read mail at the same time. A shared address would let them read each other’s codes; a fresh address per test makes the whole class of problem impossible.

Time is metered

A test that waits sixty seconds for mail is fine. A job that waits sixty seconds in every one of forty tests is forty minutes of billed runner. The waits have to be bounded, and the suite has to be sharded once it grows.

The test code these workflows run is the one from the Playwright guide or the Python guide: a fresh address, a wait with a deadline, an extractor anchored on your template. Nothing in it is CI-specific, which is the point — the runner-specific parts all live in the workflow file.

The workflow, for Playwright

One job. It starts your application with a real outbound mailer, waits for it to answer, runs the suite, and keeps the report only when something failed.

name: e2e

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 20                 # the whole job, comfortably above every wait inside it

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - run: npm ci
      - run: npx playwright install --with-deps chromium

      # Your application, started the way it runs in staging: a REAL outbound
      # mailer. Its credentials are YOUR secret; the mailbox side needs none.
      - name: Start the application
        run: npm run start:test &
        env:
          MAILER_API_KEY: ${{ secrets.MAILER_API_KEY }}
          APP_URL: http://localhost:3000

      - name: Wait for the application
        run: npx wait-on --timeout 60000 http://localhost:3000/health

      - name: Run the suite
        run: npx playwright test
        env:
          BASE_URL: http://localhost:3000

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

Three lines carry the weight. timeout-minutes: 20 is the outer bound everything else nests inside. The application is started with its real mailer credentials, because a test that reads real mail needs real mail to be sent. And the report is uploaded only on failure, with a short retention — a passing run has nothing worth keeping.

The workflow, for pytest

The same shape with the Python toolchain: start the application, wait for its health endpoint, run the suite with a per-test timeout above the mail deadline, keep the JUnit report on failure.

name: e2e

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip

      - run: pip install -r requirements.txt -r requirements-test.txt

      - name: Start the application
        run: python -m app.server &
        env:
          MAILER_API_KEY: ${{ secrets.MAILER_API_KEY }}
          APP_URL: http://localhost:8000

      - name: Wait for the application
        run: |
          for i in $(seq 1 60); do
            curl -sf http://localhost:8000/health && exit 0
            sleep 1
          done
          echo "application did not come up" >&2; exit 1

      - name: Run the suite
        run: pytest tests/e2e -q --timeout=120 --junitxml=report.xml
        env:
          BASE_URL: http://localhost:8000

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: pytest-report
          path: report.xml

--timeout=120 comes from the pytest-timeout plugin and is the per-test ceiling; the mail deadline inside the helper is sixty seconds, so a test that waits for one message and then does some browser work still fits. The health-check loop is written out rather than pulled in as an action because it is eight lines and there is nothing to get wrong in them.

The one network rule

Reading a mailbox is an outbound HTTPS request from the runner to grabmail.io. That is the entire network footprint:

• No inbound. Nothing connects to the runner. There is no webhook to receive, no SMTP server to run, no port to expose.
• No SMTP from the runner. The mail is sent by your application through its provider, over that provider’s API or SMTP endpoint — the same way it does in production. The runner never speaks SMTP itself.
• Only grabmail.io:443 to add on a runner with an egress allow-list — plus your mail provider and your package registry, which the job needed already.

      - uses: step-security/harden-runner@v2
        with:
          egress-policy: block
          allowed-endpoints: >
            grabmail.io:443
            api.your-mail-provider.example:443
            registry.npmjs.org:443

If the suite passes locally and fails in CI with a connection error from the helper, this rule is the first thing to check, and it is almost always the whole answer. A self-hosted runner in a corporate network will typically have HTTPS egress filtered by hostname; the hostname to allow is the API’s, and the request is plain HTTPS on 443.

Shards, matrix jobs and retries

Once the suite is slow enough to shard, shard it. Playwright splits a run across jobs with --shard, and because every test opens its own mailbox, the shards need nothing from each other:

  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # ... the same steps as above, then:
      - run: npx playwright test --shard=${{ matrix.shard }}/4

The same property covers the other two ways a test ends up running twice at once:

Two pull requests at the same time

Two jobs, two sets of random addresses, no overlap. The per-client ceiling on the API is 1200 requests a minute, which is twenty mailboxes polled once a second — a job polling one mailbox at a time is nowhere near it.

A retried job

A retry runs the test body again, which invents a new address again. The old mailbox still holds the old message for 5 days, and nothing reads it — the retry never sees it.

Playwright’s own retries

Same thing, one level down: each attempt runs the fixture again. Do not move the address into a beforeAll to save time; that is exactly the sharing that lets an attempt read the previous attempt’s code.

Timeouts that fit inside the job

There are four clocks and they have to nest, innermost shortest. When they do not, the failure is reported by the wrong one and points at the wrong cause.

The symptom of a broken nesting is specific: a test that waits sixty seconds inside Playwright’s default thirty-second test timeout dies at thirty seconds with a message about the test, every time, and says nothing about the mail. Set the test timeout first, then everything outward.

Reading a failure

A failed run should tell you which of three things happened — the mail never came, the wrong mail came, or the code in it was wrong — without re-running anything. Four habits make that true:

1. Log the address. The helper’s failure message names the mailbox it waited on. Print it once more at the top of the test so it is in the job log even when the assertion is somewhere else.
2. Open the mailbox by hand. Messages stay for 5 days, so /inbox/<address> on this site shows exactly what the runner saw — or did not — for the rest of the week. This is the single most useful thing about a real mailbox over a mocked one.
3. Keep the report on failure. Playwright’s trace shows the click that should have sent the mail; the JUnit file shows which test and how long it waited.
4. Check the service status before blaming the test. The status page is probed from outside every two minutes; if inbound mail was down at the time of the run, the failure was real and not yours.

Cleaning up, optionally

Everything expires after 5 days whether or not anyone deletes it, so a run that skips cleanup costs nothing. Deleting what the run read is still worth a step, because the next failure is then read against a genuinely empty mailbox. It is idempotent — deleting twice still answers 200 — so it can never fail a build on its own:

      - name: Delete what the run read
        if: always()
        run: |
          for addr in $(cat .e2e-addresses 2>/dev/null); do
            curl -sG https://grabmail.io/api/v1/mailbox --data-urlencode "address=$addr" \
            | jq -r '.messages[].id' \
            | xargs -r -I{} curl -sX DELETE -G "https://grabmail.io/api/v1/message/{}" --data-urlencode "mailbox=$addr" -o /dev/null
          done

Make it if: always() and never make it required: a cleanup that fails should be a warning in a log, not a red build.

Before you call it done

• No mailbox credential in secrets; only your application’s own mailer key.
• Outbound HTTPS to grabmail.io allowed, and nothing inbound.
• A fresh address per test, invented in the test body — safe under shards and retries.
• The four timeouts nested: deadline < test < step < job.
• The report uploaded on failure, with the address in the log.
• Cleanup as an always-run, never-required step.

That is all the runner adds. The test discipline underneath — deadline, fresh address, anchored pattern — is in testing a verification flow end to end, and the extraction rules on their own are in OTP codes in automated tests.

Questions