The Job That Usually Finds Nothing
Several jobs in the purchasing chain exist only to correct things that should not need correcting. They run every five minutes, every hour and every night — and a run that fixed nothing looks exactly like a run that had nothing to fix.
The most valuable job in a system is often the one whose output has read "0 corrected" every day for a year.
Our take
Safety nets are unglamorous and they are the difference between a chain that mostly works and one you can rely on. The purchasing chain here has several, and they share a design: each one re-derives the truth from primary records, each one is safe to run when there is nothing to do, and each one can run again without doing damage twice. The property that makes them work is idempotence — the ability to say "make this right" rather than "apply this change" — and the property that makes them hard to manage is that a successful run and an idle run produce the same output. That second one is the real cost, and it is why a safety net needs a monitor of its own rather than the reassurance of never having complained.
Three of them, and what each is catching
They sit at different points in the chain and run at different frequencies, and the frequency tells you how quickly the thing they catch matters.
| Every | What it does | What it is catching |
|---|---|---|
| Five minutes | Dispatches analysis for any sourcing request that still has unanalysed quotations | A quotation that arrived while something was down, or an analysis that never got dispatched |
| Hour | Recomputes payment totals on any order whose recorded balance disagrees with its successful payments | A payment callback that was processed but whose posting failed, or a payment row created outside the normal path |
| Night | Closes purchase orders fully paid longer ago than the organisation's window | The ordinary human failure to close finished business |
The first two are corrections. The third is housekeeping. What they have in common is that the primary path is expected to do the work, and the sweep exists because "expected to" is not the same as "always does".
Idempotence is the whole design
A safety net has to be safe to run when it is not needed, because most of the time it is not needed. That means each one is written as a statement about the desired end state rather than as an instruction to apply a change.
-
The payment sweep recomputes rather than adds
It compares the sum of successful payments against the balance recorded on the order and reposts the total. Because the posting sets the figure rather than incrementing it, a payment already reflected is reflected exactly once, however many times the sweep runs.
-
The analysis sweep over-dispatches deliberately
It sends every sourcing request with a pending quotation for analysis, without checking whether one is already under way. That is harmless because the job at the far end carries its own guards — it respects the organisation's setting and refuses to touch anything a person has already awarded.
-
The closure sweep skips what is already closed
Orders already completed or cancelled are excluded from the candidate set, so the sweep cannot re-close or re-stamp anything.
Guard at the far end, not at the dispatcher
The analysis sweep is the clearest example of a pattern worth copying. It would be possible to make the sweep clever — track what has been dispatched, avoid duplicates, reason about state. Instead the sweep is deliberately stupid and the job it dispatches is careful. That puts the guard in one place rather than two, and it means a duplicate arriving from any source, not only from this sweep, meets the same refusal.
Make the sweep simple and the work careful, not the other way round.
The cost: silence means two things
Here is the honest difficulty, and it is stated in the code itself: a run that corrects nothing looks identical to a run with nothing to correct. Both produce a quiet, successful job and a line saying zero.
Which means the ordinary signal — "we would have heard if something were wrong" — is exactly backwards for a safety net. A sweep that has silently stopped running produces the same absence of complaint as one running perfectly, and the thing it was catching accumulates unnoticed for however long that takes to discover.
The answer is a monitor on the job rather than on its findings. Every scheduled job here reports when it started, when it finished, how long it took and whether it failed, and the scheduler itself writes a heartbeat every minute. That is what makes an absent run visible, and it is a different thing entirely from the run's own output.
What a safety net is not allowed to be
Two constraints, and both are the difference between a net and a hammock.
- It must not become the primary path. If a correction sweep is routinely fixing things, the thing it is fixing is broken and the sweep is hiding it. The measure of a healthy safety net is that it finds nothing, and that only means something if somebody is counting what it finds.
- It must not make a decision the primary path would not have made. The analysis sweep dispatches work; it does not analyse. The payment sweep recomputes a total from records that already exist; it does not create a payment. A sweep that originates business facts is not a safety net, it is a second, unsupervised way of doing the work.
Making the nets legible
The sweeps run and report their own health. What is missing is the layer above — how often each one actually catches something, and a signal when that number moves.
A correction count over time
How many records each sweep fixed, per run, kept — so a rising number is a symptom rather than a line in a log.
An alert on a net that starts catching
A threshold per sweep, above which somebody hears about it, because a safety net doing real work is news.
A trace from a correction to its cause
Which order, which payment, which callback — so a corrected record can be followed back to the path that failed.
We publish scope, not dates.
Scope operational monitoringWhat AWRA OpsHub does today
- A sweep every five minutes dispatching analysis for any sourcing request that still holds an unanalysed quotation, deliberately over-dispatching because the work it dispatches carries its own guards.
- An hourly sweep comparing each order's recorded payment total against the sum of its successful payments and reposting where they disagree, using a posting that sets rather than increments so nothing is counted twice.
- A nightly sweep closing purchase orders fully paid longer ago than the organisation's window, excluding anything already completed or cancelled.
- Hourly retries for tax filings and credit notes that failed to transmit, on the same pattern.
- Every one of them declared to run once across servers and never to overlap itself.
- Cross-organisation sweeps declared as such rather than relying on console context, so the intent is stated and the unscoped-query warnings stay meaningful.
- Start, finish, duration and failure recorded for scheduled work, plus a scheduler heartbeat every minute, so an absent run is visible.
- Guards at the far end of the dispatch — an organisation's own setting, and a refusal to touch anything a person has already acted on.
More we can add to your workspace
- A count of what each sweep actually corrected, kept over time, so a net that starts catching things is visible as a trend rather than as a line in a log.
- An alert when a correction rate rises, since a safety net doing real work is a symptom of a primary path that has stopped working.
- A trace from a corrected record back to the path that failed, so the cause is investigable rather than inferred.
- A dashboard of every scheduled sweep with its last run and last finding, in one place rather than per job.
- A dry-run mode reporting what a sweep would change without changing it, for use after a configuration change.
- Deduplication in the analysis dispatcher, which would reduce queue volume where the far-end guards make it merely wasteful rather than harmful.
Where we point you to a specialist
- We will not let a safety net originate a business fact. A sweep may recompute a total from records that exist and may dispatch work the primary path should have dispatched; creating a payment, a receipt or an award from a scheduled job would be a second unsupervised route into your records with nobody's name on it.
- We will not treat a quiet sweep as evidence that the chain is healthy. Silence from a correction job means either that nothing needed fixing or that the job is no longer running, and those are opposite situations — which is why the monitoring sits on the job rather than on its findings.
- Where a sweep would have to guess at an intent — which of two conflicting records is right, what a partial figure should become — we will keep leaving it for a person. A correction is only safe when the right answer is derivable, and the moment it requires judgement it stops being a safety net.
Keeping a count of what each sweep corrects is the smallest piece here and the one that changes the most: it turns a class of invisible job into something with a trend, and a rising trend is the earliest warning a chain gives.
Five questions to ask about any correction job
Is it safe to run twice?
A good answer sounds like
Yes, and why.
What ours actually is
Yes. The payment sweep sets a total rather than adding to it, the closure sweep skips finished orders, and the analysis sweep relies on guards at the far end.
Can it create something new?
A good answer sounds like
No.
What ours actually is
No. Each one recomputes from existing records or dispatches work the primary path should have dispatched.
How would I know it stopped running?
A good answer sounds like
A monitor on the job.
What ours actually is
Scheduled work records its start, finish, duration and failures, and the scheduler writes a heartbeat every minute.
How often does it actually fix something?
A good answer sounds like
A number.
What ours actually is
Each run reports its own count; keeping those counts over time is the piece named above and is the change that would answer this properly.
What happens if two servers run it at once?
A good answer sounds like
They do not.
What ours actually is
Every sweep is declared to run on one server and never to overlap itself.
Ask what your quiet jobs are catching
Every operational system has a few of these, and almost nobody knows their numbers. If one of yours has started catching things regularly, that is the most useful early warning available and it is currently going into a log.
Talk through operational monitoringFrequently asked questions
Why dispatch analysis for requests that may already have been analysed?
Because the guard belongs at the far end. The job that does the analysis already respects the organisation's setting and refuses to touch anything a person has awarded, so a duplicate dispatch costs a moment of queue time and changes nothing. Making the dispatcher clever would put the same guard in two places, and the second copy would be the one that drifts.
Could the payment sweep double-count a payment?
No, because the posting it triggers sets the order's payment total from the sum of its successful payments rather than adding an amount to whatever is there. Running it a hundred times produces the same figure as running it once. That property is what makes it safe to run hourly against every order in the system.
If these jobs are usually finding nothing, why run them so often?
Because the cost of a sweep that finds nothing is a query, and the cost of a correction that waits is a wrong balance on a live order or a quotation nobody scored. The frequencies track how quickly each thing matters: analysis every five minutes because a buyer is waiting, payment drift hourly because a wrong balance affects a payment decision, closure nightly because it is housekeeping.
How would we know if one of these stopped working?
From the job monitor rather than from the job's output. Scheduled work records when it started, when it finished, how long it took and whether it failed, and the scheduler writes a heartbeat every minute. A silent correction job is ambiguous by nature — the whole point is that it usually has nothing to say — so the signal has to come from whether it ran at all.
Should a safety net ever be catching things regularly?
No, and that is the most useful thing to measure about it. A correction job that fixes something every day is masking a primary path that has stopped working, and the sweep is the reason nobody has noticed. Keeping a count per run so a rising trend is visible is the first item on the list above for exactly this reason.
Do these sweeps run across every organisation?
Yes, and they say so explicitly rather than relying on the fact that scheduled work has no signed-in user. That declaration matters for a reason beyond tidiness: unscoped queries are logged as warnings so they can be reviewed, and a sweep that is legitimately cross-organisation would otherwise generate noise that makes the genuine warnings harder to see.
