The canary that guards our inbound email

In June 2026, Gmail rejected roughly 73 percent of the email we forwarded into it, and nothing on our side logged an error. The copies did not bounce back to us. They did not land in spam. They vanished, for weeks, and the bug never touched a line of our code. This is the story of how we found it, why forwarding email breaks DMARC, and the probe email that now travels our real inbound path every two hours so this class of failure can never happen to us quietly again.

We run LetterDuck, an email and newsletter workspace, and inbound mail is the part we are most paranoid about. A delayed newsletter is embarrassing. A lost inbound message is unforgivable. Most of our paranoia lives in the email deliverability handbook; this piece is about the receiving side, where almost nobody points their monitoring.

The mail that vanished

Our inbound architecture is short enough to describe in one paragraph. A catch-all route sends every address on the domain to a Cloudflare Email Worker. The Worker reads the raw RFC822 message exactly once, parses it with postal-mime, and writes the thread and message to a D1 database. D1 is the source of truth: the workspace inbox reads from it, replies thread against it, nothing else is load-bearing.

During dogfooding we also mirror every inbound message into a plain Gmail mailbox as a second copy, a belt-and-suspenders habit from before the Worker existed. The mirror's original mechanism was the obvious one: Cloudflare's message.forward(), one line of code, mail shows up in Gmail.

Until it mostly didn't. We noticed threads in the workspace that had no twin in Gmail. Then more of them. When we audited the window and compared what the Worker had ingested against what Gmail had accepted, about 73 percent of the forwarded copies had never arrived. That number is our own count from the June 2026 incident, and it came with a detail that still bothers us: our logs were green the whole time. The rejection happened downstream, at Gmail's door, on a delivery attempt made by Cloudflare's infrastructure rather than ours. Nobody in that chain owed us an error report.

The primary pipeline never dropped a message. Only the safety copy was bleeding, which is the worst kind of irony: the backup channel failed in precisely the silent way backups exist to protect against.

Why forwarding breaks DMARC

The mechanism is worth understanding because it bites anyone who forwards mail, not just us.

SPF ties a message to the servers allowed to send for its envelope domain. When a forwarder passes your message along, it re-sends it from the forwarder's own servers. Gmail checks SPF against the forwarder's IP and the original sender's domain, and the check fails, because the forwarder was never in that sender's SPF record. DKIM can survive the hop when the message travels untouched, but plenty of real mail arrives unsigned, or gets its signature broken in transit.

DMARC sits on top of both: it passes only if SPF or DKIM passes and aligns with the visible From domain. A forwarded message with failed SPF and no surviving DKIM fails DMARC outright. And when the From domain publishes a strict policy, p=reject, the receiving server is instructed to refuse the message during the SMTP conversation, a hard 550, no spam folder, no second chance. The 73 percent that died were, as far as we can reconstruct, mail from domains with strict policies or broken signatures; the survivors were mostly messages whose DKIM stayed intact across the hop.

Two things worth saying plainly. First, this is not a Cloudflare bug; any forwarder hits it, and SRS (the envelope-rewriting scheme forwarders use to patch SPF) fixes the SPF check without fixing DMARC alignment, so strict policies still fail. Second, the sender never knows either. Their message reached us fine. It was our forwarded copy that died. If you want the full authentication chain in one place, we wrote up how SPF, DKIM, and DMARC fit together separately.

The fix: insert, don't forward

The way out is to stop asking Gmail's front door for permission. The Gmail API can insert a raw RFC822 message directly into a mailbox over HTTPS. There is no SMTP transaction, so there is no SPF evaluation and no DMARC verdict. The message simply appears, labeled INBOX and UNREAD, dated by its own Date header.

Our mirror now works like this, simplified from the production handler:

// Mirror an inbound message into the Gmail archive.
try {
  await insertToGmail(env, raw);   // Gmail API insert: no SMTP, no DMARC vote
  outcome = 'mirror_ok';
} catch {
  await message.forward(to);       // last resort; may still be DMARC-rejected
  outcome = 'mirror_ok';           // (or 'mirror_fail' if this throws too)
}
await logEvent(env, outcome);      // every attempt is counted, ok or not

The forward stays only as a last-resort fallback, and every attempt, success or failure, writes a row to a monitor_event table. Those rows matter later.

The fix carried a tradeoff we want to name: an API insert needs an OAuth token, and a token is a new thing that can die. We had traded a silent failure mode for a loud-if-you-listen one, which is a good trade only if something is actually listening.

One more bug the counters caught, since this is a teardown: senders retry deliveries they think failed, and early on a slow mirror could push the email handler past its time budget, so the sender redelivered and we mirrored again. Gmail once received four copies of a single DMARC report that way. The cure was deduplication on Message-ID (we synthesize a SHA-256-based ID when a sender omits the header) and skipping the mirror on any delivery we have already fully processed. Every unique message is still mirrored exactly once.

The watchdog the incident bought us

The frightening part of June was never the bug. It was the three words "no error report." Our Worker can measure everything that happens after mail reaches it, and nothing that happens before: a deleted routing rule, a DNS change, an MX record outage, a registrar hiccup. Any of those would stop mail cold, and every dashboard we own would stay green.

So we built the thing that finally closes that gap: a canary that is itself an email.

Every two hours, a Cloudflare cron trigger fires:

// wrangler.jsonc
"triggers": { "crons": ["0 */2 * * *"] }

For each domain, the watchdog sends a probe from canary@ the domain back to canary@ the same domain, through the same relay our real outbound mail uses. The subject line is __LD_CANARY__ plus a random token. That message leaves our infrastructure entirely, resolves our MX records, passes through Cloudflare Email Routing, lands in the Worker, and runs the full production parse-and-store path. It is real mail in every way that matters.

Except the last one. At the top of ingest, after the domain has resolved to an entity, one check runs before anything touches the inbox:

// inside ingestRawEmail, before threads, storage, or the mirror
if (subject.startsWith(CANARY_PREFIX)) {
  await logEvent(env, entityId, 'canary_recv', true, token);
  return { ok: true, canary: true };   // no thread, no mirror, invisible
}

The probe's receipt is recorded and the probe evaporates. No thread is created, nothing is mirrored to Gmail, no unread badge moves. A user of the workspace cannot tell canaries exist.

The next scheduled run closes the loop. It checks four things:

every 2 hours, per domain:
  1. did the LAST canary arrive?        -> problem if not seen within 60 min
  2. send a fresh canary                 -> through the real relay, real MX, real Worker
  3. reconcile the Gmail mirror (24h)    -> problem if >= 3 fails AND > 30% fail rate
  4. probe the Gmail OAuth token         -> problem if dead (mirror is down until re-auth)

  any problem  -> ONE email to the owner (6h cooldown between alerts)
  no problems  -> clear the cooldown, send nothing

The thresholds are deliberate rather than defaults, so here they are with their reasoning:

Everything lands in two small D1 tables, monitor_event and monitor_state, and a "Mail health" card in the product's Settings tab reads the same data, so the live status is one click away when an alert does arrive.

Three rules we now build monitoring by

Test the real path. A /health endpoint proves your process is up and nothing else. Our canary exercises DNS, MX resolution, Cloudflare's routing table, the Worker runtime, the MIME parser, and the database write, because it is a genuine email taking the genuine route. Anything that would eat a customer's message eats the canary first. This is the only honest answer to failures that live upstream of your code, and June taught us those are the ones that actually happen.

Absorb the probe at the last possible hop. The canary stays indistinguishable from real mail until the final step inside ingest. Absorb it earlier, at the routing layer or in a separate handler, and you shrink the surface you are testing. The check is four lines and it runs after entity resolution on purpose.

Alert on problem; silence is healthy. A daily "all systems green" email trains you to archive alerts, and an alert you archive is an alert system you no longer have. Our watchdog emails only when something is wrong, rate-limits itself to one email per six hours, and clears that cooldown the moment things recover so the next real incident alerts instantly. The companion rule: monitoring must never break the thing it watches. The event logger swallows its own errors, and the email handler never throws, because a thrown handler bounces the sender's message, and no metric is worth bouncing real mail.

Steal this pattern

None of this is proprietary and none of it needs our stack. The recipe: find the resource your service cannot self-measure (for inbound email, that is mail which never reaches you). Send a probe through the public front door on a schedule. Tag it so the last hop can recognize and absorb it. Record receipts, reconcile on the next run, and alert only on failure, with a cooldown that resets on recovery.

The bill is close to zero. Twelve probes a day ride within any sending plan's free allowance (they do count against ours, 12 of Resend's 100 free daily sends, a price we pay gladly), the event tables are a few hundred rows a week, and the cron is free. Pair the canary with external reputation monitoring like Google Postmaster Tools and you have coverage on both directions of your mail for nothing.

If you forward email anywhere today, go count what actually arrives. We thought our mirror was fine for weeks. The number was 27 percent.