Catching a silent-loss bug on the Paystack settlement path

QuickBite takes real money through Paystack, in production, on an AWS EC2 + Dockerized stack behind Nginx and Cloudflare. In a payment flow the dangerous failures are not the ones that throw — a 500 is visible and gets retried. The dangerous ones are silent: the customer is charged, the gateway reports success, and yet the order’s internal state never advances. Money moved; the system says nothing is wrong.

This class of bug lives in the seam between "the payment provider confirmed" and "our database recorded the consequence." Anything that can happen between those two facts — a dropped webhook, a non-idempotent handler, a transaction that commits the charge but not the fulfilment — is a silent loss.

The naive integration treats the synchronous checkout response as the source of truth: the client calls "verify payment," gets a success back, and the server marks the order paid in the same request. This works in the happy path and fails exactly when it matters. If the request that marks the order paid times out after the charge succeeds, the customer is charged and the order is not paid — and nothing errors loudly enough to notice.

Relying only on the client-driven verify call also means a webhook retry from the gateway can double-apply, or an out-of-order webhook can overwrite a later state with an earlier one. Without idempotency and without treating the webhook as authoritative, "it worked in testing" hides a whole family of production-only races.

The settlement path is made authoritative and idempotent: the gateway webhook — verified by signature — is what actually transitions an order, and every transition keys on the payment reference so a retried or duplicated webhook resolves to the same state instead of a second charge’s worth of fulfilment. The synchronous verify call becomes a fast-path convenience, not the source of truth.

To make silent loss loud, the invariant "every successful charge has a matching settled order" is checked continuously, not assumed. A reconciliation pass compares the gateway’s record of successful transactions against internal orders and surfaces any gap — a charge with no settled order is an alert, not a customer support ticket three days later.

The trade-off is that the flow is now asynchronous and eventually consistent: the UI has to tolerate a brief "processing" state rather than promising instant confirmation. For payments that’s the correct bias — a moment of latency in exchange for never silently losing a settlement.

The verification is the reconciliation itself: if the gateway’s successful-transaction total and the count of settled orders agree, the silent-loss window is closed; if they diverge, you’ve caught the exact failure the naive design would have hidden. Deliberately dropping and replaying webhooks in a staging environment confirms idempotency holds and that a missed webhook is recovered rather than lost.

The lesson generalizes: on any money path, treat the provider’s asynchronous, signed callback as authoritative, make every state transition idempotent, and continuously reconcile the two systems of record so that a discrepancy pages you instead of accumulating quietly.