The Alarm That Has Never Gone Off
An uptime check asks whether the website answers. It cannot tell you that the scheduled half of the system died four days ago, because a box with dead cron serves pages beautifully. The fix is an alarm triggered by silence — and the hard part is not building it, it is knowing whether it is still connected.
Nobody reports the report that did not arrive. They assume it is coming, then they assume they missed it, and by the time somebody checks it has been four days.
Every system of this kind has two halves. One answers when spoken to: somebody opens a page, submits a form, calls an endpoint. The other runs on its own — the nightly digest, the reminder, the backup, the recurring invoice, the retention sweep, the report that lands in an inbox at seven.
Almost all monitoring watches the first half. A poller asks the site whether it is alive every minute and raises an alarm when it stops answering. This is genuinely useful and it is completely blind to the second half, because a server whose scheduled tasks have stopped running serves web pages exactly as well as one whose tasks are fine. The page loads. The check is green. Nothing has run since Tuesday.
The failure is silent for as long as nobody notices that no digest arrived, no backup ran and no reminder went out.
An alarm you set off by staying quiet
The instrument for this has an old and rather grim name. A dead man's switch is a control that has to be actively held down; release it and something happens. It comes from train cabs, where the driver keeps a pedal depressed and the brakes engage if they stop.
Applied to a scheduler, the logic inverts. Instead of a monitor asking the system whether it is alive, the system tells the monitor — every minute, as one of its scheduled tasks. The monitor is configured to expect that message. When the messages stop, the monitor raises the alarm.
An uptime check
- The monitor asks; the system answers.
- Detects a server that has stopped serving.
- Cannot detect a server that serves perfectly while its scheduled work is dead.
- Green throughout the entire outage described in this article.
- Necessary. Not sufficient.
A dead man's switch
- The system tells; the monitor listens.
- Detects the absence of the message rather than the presence of a fault.
- Fires within minutes of the scheduler stopping, whatever the cause.
- Catches the case where nothing is broken except the thing that runs things.
- Costs one outbound request a minute.
That is the easy part, and most teams that think about the problem at all get this far. The interesting part is everything that has to be true for the switch to still be working six months later.
Three rules that keep a switch honest
-
Only a real scheduled run may reset the switch
This is the rule that matters most and the one most likely to be got wrong. If an engineer can run the heartbeat command by hand and that counts, then the natural reaction to a suspected outage — go and check the scheduler — silences the alarm that was about to tell you the truth. In this product the ping is skipped unless the run identifies itself as coming from the scheduler; a manual invocation records locally and deliberately does not touch the switch.
-
The ping can never fail the run
An unreachable monitor is a monitoring problem, not a scheduler problem. If a failed ping threw, an outage at the monitoring provider would take down every scheduled task on the box — the backup, the digest, the reminders — because the thing watching for failure caused one. Failures here are logged and swallowed, which is the correct direction for that trade.
-
The timeout is short and deliberate
This runs every minute. Sixty seconds of budget per minute belongs to the entire scheduler, not to its heartbeat, so the ping is given four seconds and then abandoned. On a small server the difference between a four-second timeout and a thirty-second one is the difference between a heartbeat and a queue of heartbeats.
Rule one, restated, because it is counter-intuitive
A monitoring control that a human can satisfy by hand is not a monitoring control. It reports the presence of a curious engineer, which correlates almost perfectly with the presence of a problem. The instinct when building one of these is to make the manual command do the same thing as the scheduled one, for consistency. That instinct is exactly wrong.
The question that is harder than the alarm
Now the failure this whole article is named after. Suppose the switch was configured eight months ago, and the outbound ping has been failing ever since — a URL typed with a character wrong, a monitor account that lapsed, a firewall rule added in March.
What does that look like from the inside?
It looks like this: the scheduler runs. The site is up. Every health indicator is green. No alarm has ever fired. And an alarm that has never fired is indistinguishable from an alarm that has never been needed — which is precisely the state you would expect from a system that has been working correctly for eight months.
The one alert that would fire if the box died has been silently disconnected the whole time.
So the outcome of the ping is itself recorded — whether it succeeded, when, and the error if it failed — and kept where the health page can read it. The health page can then answer a question that is different from "is the scheduler running": is the switch actually armed? Two questions, two indicators, and confusing them is how a monitoring system becomes decorative.
That record is kept for two days, and the window is chosen rather than inherited. A ping result older than two days is not a stale ping — it is a scheduler that has stopped running, and there is already a check reporting that. Two indicators, no overlap, and neither one quietly answering for the other.
Where the local record lives
Underneath the switch is a plainer thing: a note of when the scheduler last ran. It is written twice, to a cache entry and to a file on disk, and reading prefers the cache and falls back to the file.
The doubling exists because of a specific failure. A cache outage is a common and fairly minor event, and it should not make the monitoring blind — that is the pattern where an incident and the loss of visibility into it arrive together. When the cache write fails, the payload records that it failed, a warning is logged, and the file is written anyway.
| What the heartbeat records | Why it is there |
|---|---|
| When it ran, as a timestamp and an instant | The staleness check subtracts one from the other |
| Whether the run was scheduled or manual | Rule one — a manual run must not read as a live scheduler |
| Which environment | A staging heartbeat arriving in a production view is a question, not an answer |
| Which host | On more than one server, which one is still running the scheduler |
| How PHP was invoked | Distinguishes a genuine cron invocation from something else running the command |
| Whether the cache write succeeded | So a cache outage shows as a cache outage rather than as a dead scheduler |
The queue worker — the process that handles background jobs, as distinct from the scheduler that starts them on a clock — carries the same arrangement: its own cache key, its own file, and the same staleness threshold. Two independent clocks, one shared definition of too old, which is five minutes by default and adjustable.
One honest limit, for anyone running more than one server
The file half of that pair is written to the local disk of whichever machine ran it, so on a multi-server deployment the file is that machine's and not the cluster's. The cache is the shared record and the file is the belt to its braces on a single box. It is a genuine limitation of the fallback rather than of the heartbeat, and worth knowing before reading a file-sourced timestamp as though it spoke for every node.
What to take from this if you buy software rather than write it
Two things, and neither of them requires you to care about caches or cron.
The first is that "is the system up" and "is the system doing the things it does on a timer" are different questions with different answers, and most status pages answer only the first. If a scheduled report matters to you — a nightly figure, a reminder, a backup — ask specifically how its absence would be detected, and how quickly.
The second is more general and applies well beyond software. Any alarm that has never fired is in one of two states, and they look the same from outside. Silence is evidence of nothing until you have tested it. Whatever your alarms are — a monitor, a backup you have never restored from, an escalation path nobody has walked — the useful question is not whether they exist but when one of them last proved it.
Our take
A dead man's switch is a small piece of engineering with a large blind spot behind it, and the blind spot is the interesting part. The switch itself is one outbound request a minute and an hour of work. What separates a real one from a decorative one is three decisions that all point away from convenience: that a human checking by hand must not satisfy it, that it must never be allowed to break the thing it watches, and that its own connectedness has to be visible as a separate fact from the health of the system it reports on. We built the third of those because the failure it prevents — eight months of green while the wire is cut — is the one that would actually happen to us. If you take one operational habit from this article, make it the habit of asking, of every alarm you own, when it last proved it still works.
What AWRA OpsHub does today
- A scheduler heartbeat recorded every minute, carrying when it ran, whether the run was scheduled or manual, the environment, the host, how PHP was invoked, and whether its own cache write succeeded.
- An outbound dead-man's-switch ping to an external monitor, which only a genuine scheduled run may send, so checking by hand cannot silence the alarm.
- A ping that can never fail the scheduler: an unreachable monitor is logged and swallowed rather than allowed to take down the scheduled tasks it exists to watch.
- A four-second timeout on that ping by default, configurable, chosen against a one-minute budget shared with everything else the scheduler does.
- The outcome of the last ping stored separately, so a health page can distinguish "the scheduler is running" from "the switch is armed" instead of letting one imply the other.
- A two-day window on that record, deliberately matched to the heartbeat, so an older entry is reported by the staleness check rather than being misread as a stale ping.
- The heartbeat written to both a cache entry and a file, with reads preferring the cache and falling back to the file, so a cache outage does not also blind the monitoring.
- The same arrangement for the queue worker, with its own cache key and file and a shared five-minute staleness threshold.
- A staleness threshold that is configuration rather than a constant, adjustable per deployment.
- A command that records locally without pinging, for the cases where a heartbeat is wanted and a reset is not.
More we can add to your workspace
- A workspace-facing view of scheduler health, so an owner can see for themselves that the scheduled half of their system ran last night rather than inferring it from a report arriving.
- Per-task heartbeats, so a single scheduled job that has been failing quietly for a month is distinguishable from a scheduler that is running everything correctly.
- A test-fire control for the switch, letting somebody deliberately break the ping and confirm the alarm arrives, which is the only real proof an alarm works.
- A record of ping results over time rather than the last one, so an intermittent monitor shows as intermittent instead of resolving to whatever happened most recently.
- A shared heartbeat file across nodes, so the fallback speaks for the cluster rather than for whichever machine wrote it.
- Overdue detection for a scheduled report a workspace relies on, raising a notice at the moment the send was due — the same silence-watching described above, one layer closer to the person who cares about the result.
Where we point you to a specialist
- We would decline to route infrastructure alerts of this kind to a workspace owner. An alarm belongs to whoever can act on it, and paging a customer about a cron process they cannot restart converts a monitoring signal into anxiety. Our position is that the operator gets the alert and the customer gets the consequence handled.
- Choosing which external monitor to trust with a dead man's switch is a decision we think belongs with whoever runs the infrastructure, and we will point you at the question rather than shipping a default provider. The one property to insist on is independence — a monitor sharing infrastructure with the thing it watches goes quiet at the same moment.
- We would not add a health indicator that reports green while unable to verify its own connection. An indicator that cannot distinguish "working" from "disconnected" is worse than no indicator, because it is consulted and believed.
A workspace-facing view of scheduler health, and a notice when an expected scheduled report has not been produced, are the two that move this from an operator concern to something a customer can see. Both read from records that already exist.
Bringing the signal closer to the person who notices
Everything above is instrumentation for whoever runs the servers. The work worth scoping is the step that puts the same facts in front of the person whose Monday morning report did not arrive.
Scheduler health a workspace can see
When the scheduled half last ran, in the workspace's own view, rather than inferred from the presence of an email.
A notice for the report that did not arrive
Absence detection one layer up: the schedule was due, the send did not happen, and somebody is told without having to notice.
A test-fire for the switch
The only honest proof an alarm works is having watched it go off on purpose. Making that a control rather than an experiment is a small piece of work with a disproportionate effect on trust.
We publish scope, not dates.
Scope operational visibilityFour questions for any vendor about the half that runs on a timer
How would you know if scheduled tasks stopped running?
A good answer sounds like
An alarm on their absence, within minutes.
What ours actually is
A dead man's switch pinged every minute by the scheduler, with an external monitor watching for the gap. Ask this before asking about uptime — uptime is the question everybody has an answer to.
Can somebody on your team satisfy that check by hand?
A good answer sounds like
No, and here is why not.
What ours actually is
No. Only a genuine scheduled run pings; a manual invocation records locally and does not reset the switch.
How do you know the alarm itself is still connected?
A good answer sounds like
A separate indicator, checked routinely.
What ours actually is
The outcome of the last ping is stored separately from the heartbeat, so a health page can report an armed switch and a disconnected one differently. This is the question most systems have no answer to at all.
When did you last see it fire?
A good answer sounds like
A date, or a candid "we have not tested it".
What ours actually is
A fair question to put to us and to everyone else. An alarm nobody has watched go off is a plan, not a control.
Ask your own alarms when they last proved it
The monitor, the backup you have never restored from, the escalation path nobody has walked. Silence from any of them means one of two things and looks identical either way. If you would like to talk through how the scheduled half of your operations is watched, we are happy to.
Talk through monitoringFrequently asked questions
Why is an uptime check not enough?
Because it tests the half of the system that answers when spoken to. A server whose scheduled tasks have died still serves pages perfectly, so the check stays green through the entire outage. What fails is everything that happens on a timer — the digest, the backup, the reminder, the recurring invoice — and none of it announces its own absence.
What exactly is a dead man's switch here?
A message the system sends outward every minute as one of its scheduled tasks, to a monitor configured to expect it. The monitor raises an alarm when the messages stop. The logic is inverted from ordinary monitoring: nothing is detected as broken, an expected signal simply fails to arrive, which is why it catches faults that leave everything else looking healthy.
Why can a manual run not reset the switch?
Because the most likely moment for somebody to run the heartbeat command by hand is when they suspect the scheduler has stopped. If that reset the switch, investigating an outage would silence the alarm about it — and the person would then see a healthy signal they had just created themselves. The ping is therefore sent only when the run identifies itself as coming from the scheduler.
What happens if the external monitor is unreachable?
The failure is logged and the scheduler carries on. This is a deliberate trade: an unreachable monitor is a monitoring problem, and letting it throw would mean an outage at a monitoring provider took down the backups, the digests and the reminders. The visible consequence is an alarm at the monitor for a scheduler that is actually fine, which is the right way round for that mistake to happen.
How is a broken switch detected, given that it fires nothing?
By recording the outcome of the ping separately from the heartbeat, so that whether the last ping succeeded is a fact a health page can display. Without that record, a ping failing since the day it was configured is indistinguishable from one that has simply never been needed — both are silence, and both look like eight months of things going well.
Why store the heartbeat in two places?
So a cache outage does not blind the monitoring at the same moment it causes an incident. The heartbeat is written to a cache entry and to a file, and the payload notes whether the cache write succeeded, so a cache problem reports as a cache problem rather than as a dead scheduler. One caveat worth knowing on a multi-server deployment: the file is local to whichever machine wrote it, so the cache is the record that speaks for the whole cluster.
