Eleven Checks and One Verdict
One screen answers whether the machinery is working, from eleven checks reduced to a single verdict. The rule that reduces them is three lines long, and the most important thing in it is where "unknown" is counted.
The question is never "is anything wrong". It is "is anything wrong that I have to do something about right now", and those need different answers.
Our take
A health screen is only useful if it can be glanced at, and it can only be glanced at if the reduction from many signals to one verdict is defensible. This one has three states and a rule short enough to hold in your head: any critical check not in a good state means attention is needed; otherwise any warning or unknown means watch; otherwise everything is fine. The decision that makes it trustworthy is counting unknown with warning rather than with fine — because a check that could not determine its own answer is not evidence of health, and a system that treated it as such would report green for a subsystem it had lost sight of. That single choice is what separates a status page you believe from one you learn to ignore.
The eleven
They fall into four groups, and the grouping tells you what the screen is actually for.
| Group | Checks | The question it answers |
|---|---|---|
| The scheduler | The scheduler itself and its heartbeat ping | Is the timed half of the system alive at all? |
| The queue | The worker, the queue storage and the queue dashboard | Is background work being picked up and processed? |
| Backups | The database dump, the intraday runs and the file backup | If today went badly, what would we recover from? |
| Capacity and specifics | Database storage, runtime storage, and quotation analysis | Is anything filling up, and is the one time-sensitive pipeline keeping pace? |
The last group is the interesting one. Two general capacity measures alongside one named business pipeline — analysis of quotations, which runs every five minutes because a buyer is waiting for it. Including a specific business process on an infrastructure health screen is unusual and correct: the thing an operator most needs to know is not only whether the machine is running but whether the one process with somebody waiting on it is keeping up.
Every check has the same shape
A key, a human label, a status, whether it is critical, a one-line summary and a set of details. Six fields, identical across all eleven, which is what makes them reducible to a verdict at all — and what makes the screen readable, since every row says the same kinds of thing in the same order.
Four of the eleven are flagged critical. The rest are informative: worth knowing, not worth waking somebody. That flag is the only input to the first half of the verdict rule, so deciding which checks carry it is the whole of deciding what "needs attention" means.
Deciding which checks are critical is deciding what will wake somebody up.
Where unknown is counted
Each check reports one of a small set of states, and one of them is unknown — meaning the check ran and could not establish an answer. A backup mechanism that has never recorded a run reports exactly that, rather than reporting a problem or reporting health.
In the reduction, unknown is counted alongside warning. That is a deliberate choice and it is the one worth arguing about, because the alternative is superficially reasonable: no evidence of a problem, so do not raise one.
-
An unknown is a loss of visibility
Not a report of health. The subsystem may be fine or it may have stopped weeks ago, and the check is saying it cannot tell you which.
-
Treating it as fine hides the failure that matters most
The most dangerous outcome for a backup is not a failed backup — it is one nobody can confirm ever ran. Counting unknown as green makes that state indistinguishable from a working backup.
-
Counting it as watch is honest and proportionate
It does not claim a problem, and it does not claim health. It says somebody should look, which is exactly what an unknown deserves.
Backups are checked three separate ways
Not one backup check but three, and the reason is that "we have backups" is three different claims with three different failure modes.
The daily database dump is looked for on disk by name pattern, and the newest is reported with its filename and age — or an explicit statement that there is none in the expected place, naming the path it looked in. A summary that names where it looked is far more useful than one that says no backups found, because the most common cause is that they are somewhere else.
The intraday runs are separate, and where none has been recorded the check reports unknown with the expected cadence stated. Publishing the expectation next to the observation is what makes the check actionable: the reader learns both that nothing was recorded and what should have been.
And the file side is its own check, because a database dump without the documents it references is a partial recovery, and the two fail independently.
The screen is a starting point, not a destination
Alongside the checks, the snapshot carries the queue's current state, the scheduled tasks with their status and their next due time, the server-level cron definitions with evidence for each, the commands an operator would actually run, and links out to the logs, the queue dashboard, the application's own pulse page and two external observability services.
That combination is the design worth noting. A health screen that only shows verdicts sends its reader somewhere else to act, and the trip is where the diagnosis is lost. Carrying the next-due time, the exact command to run a task now, and the evidence a cron is installed means the answer to "what do I do about it" is on the same screen as the problem.
From a snapshot to a history
Eleven checks and a defensible verdict cover the question of what is true now. What operators ask for next is what was true yesterday, and to be told rather than to look.
A history of the verdict
The overall status kept over time, so a subsystem that flaps between watch and fine is visible as a pattern rather than as whatever it happened to be when somebody opened the page.
Alerting on a transition
A message when the verdict changes, rather than a screen somebody has to remember to open.
A backup restore test
Confirmation that the newest dump can actually be restored, which is the only version of a backup check that proves anything.
We publish scope, not dates.
Scope operational monitoringWhat AWRA OpsHub does today
- Eleven checks in one snapshot covering the scheduler, its heartbeat, the queue worker, queue storage, the queue dashboard, database backups, intraday backups, file backups, database storage, runtime storage and the quotation analysis pipeline.
- A uniform shape on every check — key, label, status, whether it is critical, a summary and details — which is what makes them reducible and readable together.
- A three-state verdict from a rule short enough to hold in your head: a critical check not in a good state means attention; otherwise any warning or unknown means watch; otherwise fine.
- Unknown counted with warning rather than with fine, so a check that lost sight of its subsystem never reports as health.
- Backups checked three separate ways, since a database dump, its intraday runs and the file side fail independently.
- Summaries that name where the check looked and what it expected, rather than only what it found.
- A storage check that branches on the database driver and reports rather than throws on one it does not recognise.
- The queue state, the scheduled tasks with their next due times, the server cron definitions with evidence, and the commands an operator would run, all on the same snapshot.
- Links out to the logs, the queue dashboard, the application pulse page and two configured external observability services.
- A generation timestamp on the snapshot, so a stale page is identifiable as one.
More we can add to your workspace
- A history of the verdict over time, so a subsystem flapping between states is visible as a pattern.
- Alerting on a change of verdict, rather than a screen somebody has to open.
- A restore test on the newest backup, which is the only check that proves a backup is a backup.
- Per-check thresholds configurable by the operator, rather than fixed in the check.
- A projection on the storage checks, estimating when a disk will be full rather than reporting how full it is.
- A recorded acknowledgement on a check, so a known issue being worked on is distinguishable from one nobody has seen.
Where we point you to a specialist
- We will keep counting unknown with warning rather than with fine. A check that could not establish its answer has lost sight of its subsystem, and reporting that as health makes the most dangerous state — a backup nobody can confirm ever ran — indistinguishable from the safest one.
- We will not report a backup as verified because a file exists. The check reports what it found and where it looked, which is honest; proving a backup can be restored requires restoring it, and we would rather name that as separate work than let a filename imply it.
- This screen describes the platform's own machinery. Whether your business processes are healthy is a question your own reports answer, and a green verdict here is a statement about infrastructure rather than about your operations.
A history of the verdict is the contained piece with the widest effect, because the current state is already computed and all it adds is keeping it — and a subsystem that is fine whenever anybody looks is the hardest kind of problem to catch.
Five questions to ask about a health screen
How is the overall verdict computed?
A good answer sounds like
A short, stated rule.
What ours actually is
A critical check not in a good state means attention; otherwise any warning or unknown means watch; otherwise fine.
What happens when a check cannot determine its answer?
A good answer sounds like
It counts against health.
What ours actually is
Unknown is counted with warning, because a check that has lost sight of its subsystem is not evidence of health.
Which checks are critical?
A good answer sounds like
A named subset.
What ours actually is
Four of the eleven. The flag is the only input to the first half of the verdict, so choosing it is choosing what wakes somebody.
Are backups checked?
A good answer sounds like
Yes, and how.
What ours actually is
Three ways — the daily dump by name and age, the intraday runs against a stated expected cadence, and the file side separately.
Can I act from the screen?
A good answer sounds like
Yes.
What ours actually is
Next due times, the exact command to run a task now, evidence that a cron is installed, and links to the logs and the queue dashboard.
Look at how your own status page treats unknown
It is the single most revealing question about any monitoring. A page that reports green when a check could not run is a page that will be green on the morning it matters.
Talk through monitoringFrequently asked questions
What do the three overall states mean?
Attention needed means at least one check marked critical is not in a good state. Watch means no critical failure but at least one warning or unknown. Fine means neither. The rule is deliberately short — a verdict computed by something you cannot summarise in a sentence is a verdict people stop trusting.
Why does an unknown check count against the verdict?
Because it means the check lost sight of its subsystem rather than confirming it is healthy. The most dangerous state a backup can be in is not failed but unconfirmed, and counting unknown as fine would make that indistinguishable from working. Counting it as watch claims neither health nor failure — it says somebody should look, which is what an unknown deserves.
Why are there three backup checks?
Because "we have backups" is three claims with three independent failure modes: the daily database dump, the intraday runs between them, and the files the database references. A database dump without the documents is a partial recovery, and intraday runs stopping while the nightly one continues is invisible if the two are checked together.
Does a green verdict mean our data is safe?
It means the checks found what they expected to find. A backup check reports that a dump exists with a given name and age; it does not restore it. Proving a backup is a backup requires restoring it, and that is named separately rather than implied by a filename — which is the honest position and also the one worth asking of any vendor.
Why is a business process on an infrastructure screen?
Because the quotation analysis pipeline runs every five minutes with a buyer waiting on it, and an operator needs to know whether it is keeping pace as much as whether the disk is filling. Mixing one named time-sensitive process into an infrastructure snapshot is unusual and it reflects what the person reading the screen actually needs to know.
Will we be told when something changes?
Not yet — the snapshot is computed when the screen is opened, and a daily queue health digest goes out separately. Keeping the verdict over time and alerting on a transition are the first two items on the list of what would be added, and they matter most for the subsystem that is fine whenever anybody happens to look.
