The Webhook That Arrived Twice
A customer pays once. The provider tells you three times, or tells you eleven minutes late, or does not tell you at all. Everything difficult about taking money online is in that sentence, and none of it is about the payment.
Taking a payment online is two events that look like one. The customer pays their provider, and the provider tells you about it. The first is reliable and the second is not, and almost every difficulty in this area comes from treating them as a single thing.
The provider tells you by sending a message to your system — a webhook. It is a network request from somebody else's infrastructure to yours, which means it can arrive twice, arrive out of order, arrive eleven minutes late, or fail to arrive at all while the customer stands there holding a confirmation message on their phone.
What separates a payment integration that works from one that produces a monthly reconciliation argument is entirely how it handles those four cases.
The four things that actually go wrong
Worth naming individually, because they have different causes and different fixes, and a system can handle one and not the others.
| What happens | What it looks like to you | What it costs |
|---|---|---|
| The same webhook arrives twice | One payment, recorded twice | A customer credited double, found at month end |
| It arrives very late | A payment that appears hours after the sale | Somebody chases a customer who already paid |
| It never arrives | The customer paid, your system disagrees | The worst one — you are arguing with a customer holding proof |
| It arrives and your handler fails | The provider thinks it told you; nothing recorded it | Invisible, because nothing errored on either side |
The fourth is the one people do not anticipate and the one that causes the most trouble, because both parties believe the message was delivered. The provider got a response. Your system got the message. Something in between did not finish, and nothing anywhere is flagged.
The customer paid, the provider says it told you, and your system has no record. Nothing errored on either side, which is precisely why nobody noticed.
How a duplicate is caught
Providers retry. They are supposed to — a retry is how they handle not getting a clean response from you — and it means your system will receive the same notification more than once as a matter of normal operation rather than as an error.
So the receiving side has to recognise a message it has already handled. Two mechanisms are used here, in order of preference.
-
An idempotency key, if the sender provides one
A header naming this specific delivery. If the provider sends one, that key identifies the message regardless of what else changes in the payload, which is the correct and reliable way to do this.
-
A hash of the exact body, if it does not
Where no key is sent, the message is identified by hashing the credential together with the raw request body. Two byte-identical deliveries are recognised as the same message.
-
Either way, the log is the record
The identifier is recorded against your integration credential, and a second arrival finds the existing entry rather than creating one — so the handler knows it is looking at a replay before it does anything.
-
The response is recorded too
What status was returned and what body went back, so a disputed delivery can be examined rather than reasoned about.
The gap in the fallback, stated plainly
Exact-body matching only recognises byte-identical retries. If a provider does not send an idempotency key and varies anything in its retry payload — a timestamp, a retry counter, a nonce — the fallback does not see it as a replay. That is not a hypothetical: some providers do exactly this. Before you rely on duplicate protection with a given provider, check whether they send an idempotency header, and if they do not, capture two real retries and compare the bodies byte for byte.
The webhook that silently failed
A webhook that arrives and whose handler does not finish leaves a log entry stuck in a processing state. Nothing raises an error, because from the provider's point of view the message was delivered and from yours it was received.
These are caught by time. An entry still processing after ten minutes is almost certainly stuck rather than slow, and gets flagged as errored so it becomes visible rather than remaining as a row nobody looks at. Entries that have already been retried several times are left alone rather than flagged repeatedly.
Separately, there is the case where the payment was recorded but never reached the ledger — the transaction is marked successful, it has something to post against, and the posting did not happen. Those can be re-posted, which recovers a payment that is in the system but not in the accounts.
And the part somebody has to do
The recovery for both of those is a retry action, and it is worth being precise about how it runs, because this determines what your process needs to cover.
It runs when somebody presses it. It is not on a schedule. So a payment that succeeded with the provider and failed to reach your ledger will sit in that state indefinitely until a person opens the webhook log and triggers a retry.
It also looks back thirty days and handles a hundred at a time. The batch limit is sensible. The window is the one to plan around: a payment that failed to post and was not noticed for a month cannot be recovered through that route at all, and becomes a manual journal.
What AWRA OpsHub does today
- Duplicate detection, preferring an idempotency header from the provider and falling back to an exact hash of the credential and the request body.
- Every delivery logged with its identifier, the response status returned and the body sent back, so a disputed delivery is examinable.
- Stuck deliveries flagged by time — anything still processing after ten minutes is marked as errored so it becomes visible.
- A retry that re-posts successful payments which never reached the ledger, recovering money that is in the system but not in the accounts.
- A visible webhook log, so this is a screen somebody can be asked to check rather than a black box.
What it does not do
- The retry does not run on a schedule. It runs when a person presses it. A payment that failed to post sits in that state until somebody looks, which makes checking the log a process rather than a convenience.
- The retry window is thirty days. A failure not noticed within a month cannot be recovered that way and becomes a manual journal entry.
- Exact-body matching is the fallback. Where a provider sends no idempotency key and varies its retry payload, a duplicate is not recognised as one.
- No alert when a webhook goes stuck. Entries are flagged in the log; nothing emails anybody, so the log has to be opened.
Not ours, by choice
- We will not treat a provider's message as proof on its own where we can reconcile instead. A payment that exists only because a webhook said so, with nothing to check it against, is a figure with one source.
- We will not silently swallow a delivery we could not process. A stuck entry is flagged as an error even though flagging it makes the log look worse, because a clean log that is hiding failures is the more expensive kind of tidy.
Scheduling the retry, alerting on stuck deliveries, and widening or configuring the recovery window, are all scope rather than ceilings — the job, the log, the flagging and the notification infrastructure all exist and work.
Until the retry is scheduled, the single most valuable habit here is a person opening the webhook log on a fixed day each week. It takes two minutes, and the failure it catches — a payment received by the provider and never posted to your ledger — is one that reconciles to nothing and is found by a customer rather than by you.
What to do about it operationally
None of this needs a technical response from you. It needs a short, boring routine that somebody owns.
The weekly payment integration check
- Open the webhook log and look for entries flagged as errored. Anything there is a delivery that did not complete.
- Trigger the retry if there are any, and check the same entries afterwards rather than assuming.
- Compare the provider's own statement of settled payments against what is in your system for the week. This is the only check that catches a webhook that never arrived at all.
- Watch for payments recorded twice against the same customer on the same day, particularly with a provider that does not send an idempotency key.
- Check nothing is sitting near the thirty-day edge, since the retry cannot reach past it.
The third item is the one that cannot be automated away and is the most important. A webhook that never arrives leaves nothing behind — no log entry, no error, no stuck row — so the only way to find it is to compare your record against the provider's. Weekly is usually enough; daily if your volumes are high or your margins are thin.
Why this is worth understanding rather than delegating
Payment integrations are usually treated as plumbing — something that either works or does not, and either way is somebody else's department. The reason to understand the failure modes is that they do not present as failures.
A duplicated payment presents as a customer with a credit balance. A missing one presents as a customer who says they paid. A payment that never posted to the ledger presents as a reconciliation that is out by an amount nobody can trace. In each case the symptom appears somewhere unrelated, days later, and is investigated as an accounting problem rather than an integration one.
Knowing the four shapes means recognising the symptom, which converts a two-day investigation into opening one screen.
The collection side is in receivables and collections, the daily reconciliation habit in the retail daily close, and the wider chain in sales software in Kenya.
Our take
Give one person the webhook log and a fixed day each week. Duplicate detection, delivery logging and stuck-entry flagging all work, and the retry that recovers an unposted payment works — but it runs when somebody presses it rather than on a schedule, and it only reaches back thirty days. So the whole thing rests on somebody looking. And whatever else you do, compare your week against the provider's own settlement statement, because a webhook that never arrived leaves nothing behind at all and that comparison is the only place it shows up.
See the payment integration log
Duplicate detection with an idempotency key where the provider sends one, every delivery logged with its response, stuck entries flagged by time, and a retry that re-posts payments which never reached the ledger.
Explore paymentsFrequently asked questions
Can a customer be recorded as paying twice?
It is guarded against. Where the provider sends an idempotency header, that key identifies the delivery and a retry is recognised regardless of what else changes in the payload. Where it does not, the message is identified by hashing the credential and the exact request body, so byte-identical retries are caught. The gap is a provider that sends no key and varies its retry payload — a timestamp or a retry counter — because the fallback then sees a different message. Check which category your provider is in before relying on it.
What happens if a webhook never arrives?
Nothing, which is exactly the problem — there is no log entry, no error and no stuck row, because nothing reached you. The customer has paid, the provider believes it told you, and your system disagrees with both. The only way to find these is to compare the provider's own settlement statement against your records on a regular rhythm. Weekly is usually enough; daily if your volumes are high. No amount of handling on the receiving side can detect a message that was never received.
What is a stuck webhook?
A delivery that arrived and whose handler did not finish, so the log entry sits in a processing state with nothing errored on either side. These are caught by time — anything still processing after ten minutes is almost certainly stuck rather than slow, and gets flagged as an error so it becomes visible. Entries already retried several times are left rather than flagged repeatedly, which keeps the log readable.
Are failed payments retried automatically?
No. There is a retry that finds successful payments which never reached the ledger and re-posts them, and it runs when somebody presses it rather than on a schedule. That makes opening the webhook log a process rather than a convenience — give it to a named person on a fixed day. It also looks back thirty days, so a failure that goes unnoticed for a month cannot be recovered that way and becomes a manual journal.
Why would a payment be in the system but not in the accounts?
Because recording the payment and posting it to the ledger are two steps, and the second can fail on its own. The transaction shows as successful, it has an invoice to post against, and the posting did not happen — so the payment is visible on the customer and absent from your accounts. It presents later as a reconciliation that is out by an amount nobody can trace, which is investigated as an accounting problem rather than an integration one. The retry recovers exactly this case.
Is this something our accountant should be checking?
Somebody should, and it matters less who than that it is a fixed habit. The four failure modes all present as accounting symptoms days later — a customer with an unexplained credit, a customer insisting they paid, a reconciliation out by an untraceable amount — and each is investigated from the wrong end. Two minutes in the webhook log on a Monday, plus a weekly comparison against the provider's settlement statement, catches all four while they are still small.
