Payment integrations are demonstrated on the happy path and lived on every other one. Everything below is the part that does not appear in a sales deck and determines whether you sleep.
The timeout problem
You send a request to create a payout. The connection drops before the response arrives. You now do not know whether money moved.
There are two bad options — assume it failed and retry, risking a double payment; or assume it worked and do nothing, risking an unpaid recipient — and one good one.
The good one is an idempotency key: a value you generate and attach to the request. If the operation already happened under that key, the server returns the original result instead of creating a second one. Retrying becomes safe, and the ambiguity disappears.
Two things to get right. The key must be stable across retries — generating a new one each attempt defeats the purpose entirely. And it should be derived from something meaningful in your own system, an invoice line or a payout row, so that the same business event always produces the same key no matter which of your processes sends it.
Webhooks arrive more than once
Any delivery system that retries will eventually deliver the same event twice. The network hiccups after your server has processed the message but before the acknowledgement lands, and the sender quite correctly tries again.
So handlers must be idempotent too. Key on the operation identifier and the status together, record what you have already applied, and discard repeats silently. A handler that increments a counter or appends a ledger line without this check will produce beautifully wrong numbers.
Events can also arrive out of order. Treat the status you hold as a state machine and refuse backwards transitions rather than overwriting.
Verify the signature, and verify the time
A signed webhook proves the message came from who it claims. It does not prove it is current — a captured message can be replayed later, and it will verify perfectly.
That is why the signature covers a timestamp as well as the body. Check both: reject anything older than a few minutes, and compare signatures in constant time so the comparison itself does not leak information.
Deliveries stop eventually
Retry schedules are finite. If your endpoint is down long enough, attempts stop and the events are gone.
Which means a webhook handler alone is not a complete integration. You need a reconciliation path — a way to ask directly for the current state of anything you might have missed — and something that runs it after every outage. The version of this that actually works is scheduled, not manual, because manual means “when someone remembers”.
Encryption is not only about privacy
Where request bodies are encrypted, a single-use nonce does more than scramble the contents. It stops the same envelope being replayed, which is a different attack from reading it.
Generate nonces from a cryptographic source, never from a clock or a counter. A nonce derived from the current time is predictable, and predictable defeats the entire purpose.
Keep production and test apart by construction
The safest arrangement is that the credential decides the environment, not the URL. Then a test key cannot touch production balances even if someone points it at the wrong host, and a production key is refused by test endpoints. Configuration mistakes stop being dangerous.
The short version
Stable idempotency keys derived from your own records. Handlers that tolerate duplicates and out-of-order arrival. Signature and timestamp both checked. A scheduled reconciliation job you have actually tested by turning your endpoint off. Do these five and the three-in-the-morning calls mostly stop.