Exactly-once push across FCM and APNs with an outbox + dedup key

Cargoland is three tightly-coupled apps — consumer, vendor, rider — kept in lockstep, and order alerts drive the whole marketplace: a vendor has to hear a new order, a rider has to hear an assignment. Those alerts go out over two different transports, FCM for Android and APNs for iOS, each of which is at-least-once by design. Neither platform promises a message arrives exactly once, and both will happily redeliver.

So the requirement — "the vendor sees each order exactly once" — cannot be met by the transport. It has to be met by the system on top of it.

The naive path fires a push inline with the event that caused it: when an order is created, the request handler calls FCM/APNs directly and moves on. Two things break. First, the send is now coupled to the request — if the push provider is slow or down, the order request is slow or fails, and a retry of that request sends the alert again. Second, there is no record that "this order’s alert was sent," so a retry, a redeploy mid-request, or a duplicate event all produce duplicate buzzes on the vendor’s phone.

Trying to fix this by making the send "best effort and fire-and-forget" trades duplicates for lost alerts — now a transient provider blip means the vendor never hears about an order at all, which in a delivery marketplace is worse.

The design splits the decision to notify from the act of notifying, using the transactional outbox pattern. When an order is created, the notification intent is written to an outbox table in the same database transaction as the order itself — so either both commit or neither does, and there is no window where an order exists without its pending alert (or vice versa). A separate worker drains the outbox and calls FCM/APNs.

Because the transports are at-least-once, the worker is built to expect redelivery: each notification carries a stable dedup key derived from the event (e.g. order id + channel), and both the send-side bookkeeping and the client collapse repeated deliveries of the same key into one. At-least-once transport plus idempotent, dedup-keyed handling gives the effect of exactly-once where it’s observable — on the vendor’s screen.

The trade-offs: an outbox adds a write and a background worker, and delivery is asynchronous, so there’s a small, bounded delay between the order committing and the phone buzzing. In exchange you get a guarantee that survives provider outages, request retries, and redeploys — the failure modes that actually happen in production.

The properties to test are the ones the naive version fails: kill the push worker mid-drain and confirm the outbox row is still pending and gets delivered on restart (nothing lost); force the transport to redeliver and confirm the dedup key collapses it to a single visible alert (nothing duplicated); and roll back the order transaction to confirm no orphan notification was sent.

The same shape — write intent transactionally, drain it with an idempotent, dedup-keyed worker — is exactly the reliability spine behind the standalone Distributed Notification Service, which is why building one from the ground up is a deliberate study rather than a detour.