What Our Status Page Calls Down
A status page that someone has to remember to update says “All systems operational” through most outages. Ours is sampled every five minutes, per component, and a typed incident can make the page look worse than the measurement but never better. Here is what each component has to do to count as up.
A finance manager in Mombasa cannot get the payroll export to arrive by email. She opens our status page, which says everything is operational, and spends an hour assuming the fault is hers. The page was not lying, exactly. Nobody had told it anything. That is the normal condition of a status page that is typed rather than measured: it is accurate when someone is watching and has time to update it, which during an incident is the one time they do not.
So our status page is not typed. Every five minutes a scheduled task checks five components once each and adds the result to that day’s tally. The bars, the uptime percentages and the current state are all worked out from those tallies. People can add an incident, with updates in plain words, but the rules for how an incident combines with the measurement are fixed, and they only run in one direction.
Five components, five definitions of up
“Is it up?” has a different answer for each part of the system, because each part fails differently. A server can answer web requests while the queue that sends email is stuck. The question has to be asked separately, in a form each part can actually fail.
| Component | The check | Counts as down when |
|---|---|---|
| Web app & API | A request to the app’s health route, from the server to itself | It errors, times out after 8 seconds, or returns anything but success. Over 3 seconds is degraded |
| Database | A trivial query | The query fails |
| Cache & sessions | Write a random value, read it back | The value does not come back |
| Background jobs | A heartbeat job queued every minute | No heartbeat has been processed in the last 5 minutes |
| Scheduled tasks | The scheduler’s own heartbeat | A sample that should have arrived did not. A heartbeat older than 5 minutes is degraded |
The finance manager’s problem was in the fourth row. Payroll exports, emails and notifications all run as background jobs; a stuck worker leaves the web app answering perfectly while nothing that happens after you click arrives. Measured that way, the page shows Background jobs down and Web app & API up, which is both true and the thing she needed to know. What each of those jobs did, and why it failed, is on its own record, described in every background job leaves a row.
Why the scheduler is counted differently
The sampler itself runs from the scheduler. If the scheduler stops, no samples are recorded at all, and a component with no samples looks like a quiet day. So for that one component, a sample that was expected and never came is counted as down: its absence is the outage. For the other four, a gap means no data, shown in grey, because charging the web app for the scheduler’s outage would make both figures wrong. We wrote about alarms that work by silence in the alarm that has never gone off.
From samples to a bar
Each component has ninety bars, one a day. A day’s uptime is the share of that day’s samples that answered, with slow-but-answering counted as up. A day at 99.9 per cent or better is a full bar; from 95 per cent it is partial; below that it is down. A day with no samples is grey, and the page says plainly that grey means nothing was measured, not that it was fine. The figure beside each component is its uptime over the ninety days.
One bad afternoon for Background jobs
Below 99.9 per cent and above 95: a partial bar. The other four components that day stay full.
The current state at the top of each component comes from its latest sample, as long as that sample is less than fifteen minutes old. If it is older, the component shows No recent data rather than its last good reading, except the scheduler, where staleness is itself the outage. The overall line at the top of the page is the worst of the five.
An incident can make the page look worse than the measurement. It can never make it look better.
What a typed incident is allowed to do
Some things a probe cannot see: a payment provider rejecting our callbacks, a bug in one report, an email provider delaying delivery. For those our team posts an incident, against one component or all of them, with a severity and a running series of updates: investigating, identified, monitoring, resolved. Resolved incidents keep their duration and any postmortem, and stay on the page for the ninety days.
A minor incident is open
Held at degraded at least
If the measurement is worse, the measurement shows.
A major incident is open
Held at partial outage at least
A component the probe sees as fully down still shows major outage.
A critical incident is open
Major outage
The worst state there is, whatever the probe says.
Planned maintenance is open
Under maintenance
Only if the measurement is clean. Maintenance never hides a measured fault.
That last row is the one that matters. A maintenance window is the classic way a typed page covers an outage: declare maintenance, and whatever breaks inside it is “planned”. Here, if a component fails its check during maintenance, it shows the failure.
The same numbers, three places
Everything on the page is also published as status.json, readable from any origin, for anyone who wants our status inside their own monitoring. And the uptime figure on our service level agreement is the Web app & API series from this page, not a separate calculation: a target of 99.9 per cent a month, with a service credit of 10, 25 or 50 per cent when a month falls below 99.9, 99 or 95. The published promise and the measurement it is judged by cannot drift apart, because they are the same rows.
What AWRA OpsHub does today
- Five components sampled every five minutes, each against its own definition of up, degraded and down.
- Ninety days of daily bars per component, with days that were not measured shown as no data rather than as up.
- A scheduler outage counted from missing samples, so the one failure that stops the sampler still shows.
- Incidents that can raise a component’s state but never lower it, with maintenance shown only over a clean reading.
- Incident updates, durations and postmortems on the page for ninety days.
- A JSON twin of the page and an SLA page that reads the same uptime series.
More we can add to your workspace
- A probe from outside our own server, published on the page, for the case where the whole server is unreachable. Commission it
- Email and SMS updates for an incident, for customers who would rather be told than check. Commission it
- The services we depend on as components: mobile money callbacks, KRA eTIMS and email delivery. Commission it
Where we point you to a specialist
- We will not mark a component operational by hand. A person can declare that something is worse than the checks show; only the checks can say it is fine.
The outside probe is the extension we would add first, because the page already knows how to combine more than one reading of the same component.
A status page that sees further
The sampler, the daily rollup, the incident rules and the JSON feed are in place. Customers who rely on the page ask for these next.
An outside probe
A check from another network, shown beside the server’s own, so an unreachable server is visible on the page.
Incident subscriptions
Updates by email or SMS to the people who need to know, as each one is posted.
Your dependencies as components
Mobile money, eTIMS and your email provider measured and shown like the rest.
Three questions for any vendor’s status page
Is the page updated by a check or by a person?
What you will probably hear
Our team updates it during incidents.
How to read it
Then it is accurate when someone has time, which is not during an incident.
Does a stuck email queue show as an outage?
What you will probably hear
The site is up, so it is green.
How to read it
Then the page measures the front door. Ask what it says when the back office stops.
What does a day with no data look like?
What you will probably hear
Green.
How to read it
An unmeasured day shown as up turns every gap in monitoring into uptime.
The short version
Measure every component on its own terms, every few minutes, and show a day nobody measured as exactly that. Let people add what the checks cannot see, but only ever in the worse direction. A status page built that way is less flattering than one somebody types, and it is the only kind worth opening at four in the afternoon when the payroll export has not arrived.
See the status page
Five components, ninety days of measured history, open incidents and their updates, and the JSON feed behind them.
Open the status pageFrequently asked questions
How often is the AWRA status page measured?
Every five minutes. Each of five components is checked once and the result is added to that day’s tally, which the bars and percentages are worked out from.
What counts as down for the web app?
A request to the app’s health route that errors, takes longer than eight seconds or returns anything other than success. A response slower than three seconds counts as degraded, which still counts as available.
What does a grey bar mean?
No samples were taken that day. It does not mean the component was up. The scheduler is the exception: for it, a missing sample is counted as down.
Can an incident make the page look better than the checks?
No. An incident can hold a component at a worse state than the checks show, but never a better one, and planned maintenance is only shown over a clean reading.
Is there a machine-readable version?
Yes. Everything on the page is published as status.json, readable from any origin and refreshed every minute.