Outbound Webhooks: Signed, Retried, and Kept When They Fail
The direction most vendors do badly: something happens here and something has to happen somewhere else. Eight system events you can subscribe to, workflow rules for everything else, signed timestamps, retries that back off, and a dead-letter list — and why the dead letter is the part that matters.
Outbound webhooks are where integration quality actually shows. Sending an HTTP request when something happens is a morning's work. Sending it reliably — signed so the receiver can trust it, with an idempotency key so a retry does not duplicate, retried on failure, and kept somewhere visible when it eventually fails for good — is the difference between a feature and a liability.
The liability version is common and easy to spot: it fires, and if the receiver was down, the event is gone. Nobody notices for three weeks, because the failure is silent by construction.
What actually goes out
The request a connector sends
Header or element What the receiver gets
A JSON body you shaped yourself Payload mapping, so the field names are the receiver's, not ours
A connector-type header Which connector sent it, if you run several
An event-type header What happened, so the receiver can route without parsing the body
A timestamp header The moment it was signed — the receiver can reject anything stale
An HMAC-SHA256 signature over timestamp and body Proof of origin and integrity, and it is not replayable at leisure
An idempotency key So a retried delivery can be recognised and ignored by the receiver
Any auth headers you configured For receivers that want a bearer token or an API key of their own
The signature covers the timestamp and the body rather than the body alone. That is deliberate: signing only the payload lets a captured request be replayed forever, because the signature stays valid. Including the timestamp means the receiver can enforce a freshness window and a stolen request expires.
Payload mapping, which is the feature people underestimate
A connector fired by a workflow sends the shape you define, not a fixed internal structure. This is the difference between an outbound webhook that works with the system you already have and one that requires somebody to build a translator in the middle. A subscribed system event is the other way round: it arrives in one fixed envelope — the event name, when it happened, and the record — so a developer can write against it once.
It also means the receiving end can be something dumb. A Google Apps Script, a small endpoint on your website, a channel in a chat tool — none of them need to understand our data model, because the payload arrives already speaking their language. If you are choosing between integration approaches, this is usually the cheapest one that still counts as real integration.
Retries, and the dead letter
A connector has a retry policy: a number of attempts and a wait between them. A subscribed event is retried from the queue, the wait doubling after each failure — capped at an hour between attempts and at eight attempts whatever the policy says — so a receiver that is down for twenty minutes still gets the event. A timeout, a 5xx or a 429 is retried; a receiver that answers with any other 4xx has said no, and asking again would get the same answer. Success and failure are recorded against the connector so its health is visible. If every attempt fails, the delivery is written to a dead-letter list rather than discarded.
What happens when the receiver is down
Most SME systems discard a failed webhook and log a line nobody reads. Keeping the payload is what makes the difference between "the integration broke last month and we lost the events" and "here are the eleven events that did not arrive".
Check the connector health, not the logs
Because success and failure are recorded against the connector itself, a connector that stopped working three weeks ago is visible without anybody reading a log file. Put it in the monthly rhythm — it takes ten seconds and it is the only thing that catches a silently dead integration.
How a connector gets triggered
Two ways. A connector can subscribe to a system event — an invoice created, an invoice paid in full, a purchase order created, a customer or a vendor added, an item falling to its reorder point, or a stock transfer completed (the exact event names) — and it is sent wherever that thing happens: the web form, the API, an import or the mobile app. Or it can hang off a workflow action, which means the trigger is one of your own rules: you decide what constitutes an event worth sending, using the same conditions you would use to route an approval or raise an alert. That second route arrives as its own workflow event, and it is the one that carries your payload mapping.
Either way the address must be https, and an address that resolves to a private, internal or loopback network is refused — when the connector is saved, and again at the moment of sending, against the address actually connected to. Redirects are not followed. A webhook is a request our server makes on your instruction, and those checks are what stop it being pointed back inside a network.
-
Define the rule first, without the connector
If one of the eight system events is the thing you mean, subscribe to it and skip this step. Otherwise get the workflow condition right and have it raise a notification you can see. A rule that fires on the wrong events will send those wrong events perfectly reliably.
-
Build the receiver and make it accept anything
First version returns 200 and writes the body somewhere you can read it. Do not validate yet — you are trying to see what arrives.
-
Send the test payload
The connector can be tested with a sample payload before it is live. Use it. Look at the headers as well as the body.
-
Shape the mapping to what the receiver wants
Now that you can see both ends. This is the step that would otherwise become a translator service nobody maintains.
-
Verify the signature on the receiving side
Recompute the HMAC over the timestamp and body with your secret and compare. If you skip this, you have built an endpoint anyone can post to, and its URL will end up in a screenshot eventually.
-
Then enforce a freshness window
Reject anything with a timestamp older than a few minutes. This is what the timestamp in the signature is for and it is two lines of code.
The simpler cousin: notification channels
Not every outbound need is a webhook. If the requirement is "somebody should be told", the notification channels are the better answer: paste a webhook URL from Teams, Discord, Google Chat or Slack, or connect a chat tool, and events land where people already look. No mapping, no signature, no receiver to build. Chat alerts covers those.
Use a connector when a system has to act. Use a channel when a person has to know. Conflating the two produces either a chat room full of machine noise or an integration nobody can debug.
What AWRA OpsHub does today
- Per-connector payload mapping, so the body speaks the receiver's field names.
- HMAC-SHA256 signing over timestamp plus body, with configurable auth headers alongside for receivers that want their own token.
- An idempotency key generated per delivery and sent as a header.
- A configurable retry policy, with each attempt logged as its own call record. Subscribed events retry from the queue with a wait that doubles each time, capped at an hour and at eight attempts.
- A dead-letter record holding the payload, headers and idempotency key when the policy is exhausted, and a failed workflow delivery re-sent from the workflow log with the same body and idempotency key.
- Eight subscribable events: invoice created, invoice paid, purchase order created, customer added, vendor added, item at its reorder point, stock transfer completed, and the workflow action event.
- https-only addresses, with private, internal and loopback addresses refused at save and at send, the checked address pinned for the connection, and redirects not followed.
- Success and failure recorded against the connector, so a dead integration is visible.
- A test send with a sample payload before going live.
More we can add to your workspace
- An automatic re-drive of dead letters — one action that flushes the list once the receiver is back, in place of a decision per delivery.
- More events in the catalogue — a payment recorded, a bill approved, stock received — each raised where the thing happens, as the eight are.
- Payload mapping on subscribed events, which arrive in the fixed envelope today; mapping applies to workflow deliveries.
- A signature verification helper library for the receiving side. You compute the HMAC yourself, which is four lines in any language.
Anything above that you need, we can build for you
Everything listed above as something we can add describes what ships in the standard product today — it is a starting point, not a limit on what AWRA OpsHub can do for your organisation. Kenya's eTIMS integration and its maintained payroll engine are both in the product because clients needed them and commissioned them; neither appeared by itself, and the same door is open for whatever you just read about. One qualification so this is worth what it claims: a small number of things on this blog we deliberately leave to a specialist rather than build — a statutory ledger we will not sign our name to, a rule that would decide a tax question for you, a clinical or member-funds record that belongs in a regulated system — and where that is true the post says so in those words. Everything else is a scope, a timeline and a price.
The operational work, which is what most commissions actually are
An extra approval stage in a chain that does not match the standard one, a custom field set on employees or assets that only your sector needs, an expiry that has to block an order rather than send an email, a report your board asks for in a shape nothing produces, or a scanner or weighbridge feeding the goods-in door. These are the commissions we are asked for most often and the smallest ones we quote — and unlike a revenue-authority pipeline, none of them waits on a regulator.
The module-shaped additions, which are the ones readers ask for most often
A price list with real discount authority, a customer-facing quotation that expires, a bill of materials or recipe costing, a staff advance that is issued, acquitted and chased, a member or unit ledger, a matching rule that holds a payment. Each of these is a build rather than a setting, and each has been quoted before — a bigger piece of work than a custom field, with a written spec and a date instead of a roadmap slide.
The report, document or pack nothing currently produces
The board pack in the shape your board actually asks for, a donor or funder layout, an invoice or receipt template carrying what your regulator or your customer expects, a dataset the report builder cannot reach yet. Usually the fastest thing on this list to deliver, because the data is already in the system.
Systems, rails and hardware you already run
The accounting package, CRM, online store, core banking or custom database you intend to keep — connected through our API so a fact is entered once and appears everywhere it is needed. Plus the physical edge: a scanner, a scale, a weighbridge or a till peripheral feeding the door it belongs to.
How it works: you describe the requirement, we return a written scope, timeline and cost, and once agreed it is built into your environment and maintained as part of the product. Nothing here waits on a regulator or a published specification, which is why operational builds are the ones we quote fastest. Tell us the requirement that would otherwise rule us out — that is a better first conversation than a demo.
Tell us what your operation needsOur take
Shape the payload to the receiver rather than building a translator, verify the signature and enforce a freshness window on the receiving side, and check connector health monthly rather than reading logs. The dead-letter record is the feature to value most when comparing vendors — ask any of them what happens to a webhook after the last retry fails, and listen for whether the payload survives.
Signed, retried, and kept when it fails
Payload mapping to the receiver's field names, HMAC signing over timestamp and body, an idempotency key per delivery, a retry policy, and a dead-letter record you can actually open.
See the integrationsFrequently asked questions
What does an outbound webhook send?
A JSON body — a fixed envelope of event, time and record for a subscribed event, or a shape you define through payload mapping for a workflow action — plus headers naming the event and the delivery, a timestamp, an HMAC-SHA256 signature over the timestamp and body, and, for a workflow connector, an idempotency key and any auth headers you configured. The signature covers the timestamp as well as the body specifically so a captured request cannot be replayed indefinitely.
What happens if the receiving system is down?
Each attempt is logged individually and the configured retry policy runs — for a subscribed event from the queue, with the wait doubling after each failure up to an hour between attempts. When the policy is exhausted the delivery is written to a dead-letter record holding the payload and headers. The connector itself is marked with the failure, so a broken integration is visible rather than merely quiet. Re-sending is a decision per delivery; a bulk re-drive is something we can add.
Can we control the shape of the payload?
Yes, per connector. That is the feature most worth having, because it means the receiving end can be something simple — a small endpoint on your website, a Google Apps Script — that never needs to understand our data model. Without mapping, every integration needs a translator in the middle that somebody has to maintain.
What triggers an outbound webhook?
Either a system event the connector subscribes to — invoice created or paid, purchase order created, customer or vendor created, an item falling to its reorder point, or a stock transfer completing — or a workflow action, where the trigger is one of your own rules. The events are a published list a developer can write against; the workflow route covers everything else, with the payload shaped to the receiver.
How should the receiving end verify a request?
Recompute the HMAC-SHA256 over the timestamp and the raw body using your shared secret and compare it to the signature header, then reject anything whose timestamp is older than a few minutes. Both steps are a handful of lines in any language, and skipping them leaves you with an endpoint anyone can post to — whose URL will end up in a screenshot eventually.
When should we use a chat notification instead of a connector?
Use a notification channel when a person needs to know, and a connector when a system needs to act. Channels are a pasted webhook URL from Teams, Discord, Google Chat or Slack with no mapping, no signature and no receiver to build. Conflating the two gives you either a chat room full of machine noise or an integration nobody can debug.
