Every Background Job Leaves a Row
Every background job in this product writes a row when it starts and updates it when it finishes, with its runtime, its queue, its attempt count and — when it fails — the exception class and message. The monitor is also written so that it can never break the job it is watching.
Background work is where a system spends most of its effort and produces none of its evidence. Unless somebody arranges otherwise.
Reports, emails, imports, synchronisations, purges, notifications — a large proportion of what an operations product does happens on a queue, out of sight of any request. If none of it is recorded, the only signal that anything went wrong is somebody eventually noticing an absence.
What is recorded, and when
Three moments in the life of a job produce a record: it started, it finished, or it failed.
| Field | What it tells you |
|---|---|
| Display name | Which job this was, in readable form rather than as a class reference. |
| Queue and connection | Where it ran — which matters when work is separated across queues by priority or by kind. |
| Attempts | How many times this job has been tried, which distinguishes a first failure from a persistent one. |
| Started and finished | The two timestamps, from which everything else about duration follows. |
| Runtime in milliseconds | Computed from the start row when one exists, and left empty rather than guessed when it does not. |
| Exception class and message | On a failure, the type and the first thousand characters of the message. |
| Failed job identifier | Set only on a failure, linking the record to the framework's own failed job entry. |
Runtime is empty rather than wrong
A completion that cannot find its matching start row records no runtime instead of inventing one from an assumed start. That is the same discipline the attendance module applies to a clock-out with no clock-in, and for the same reason: an absent duration is visibly unknown, and a computed one is indistinguishable from a measured one. It also means a terminal row is still created when no start was recorded, so the completion itself is never lost.
The monitor cannot break the job
This is the property that matters most and it is easy to get wrong. A monitor that throws inside a job handler turns an observability feature into a source of failures — and worse, into a source of failures in exactly the jobs that were already having trouble.
So every handler is wrapped, and a failure to record produces a warning in the log rather than an exception. The job proceeds. The observation is lost, which is the correct thing to lose.
-
The existence check runs once
Whether the storage table exists is asked once per instance and remembered. Before it exists — during a first deployment, or a migration that has not run — monitoring quietly does nothing rather than failing every job in the queue.
-
Payload reading is defensive
Not every queue implementation exposes its payload the same way, so the monitor decodes the raw body when it has to and treats anything unreadable as empty. A job on an unusual queue driver is recorded with less detail rather than not running.
-
Every stored string is length-capped
Identifiers, queue names and display names are all truncated to what the storage accepts, and an exception message to a thousand characters. A monitor that failed on an over-long field would fail on precisely the jobs with the most interesting errors.
An observer that can break what it observes is not an observer.
What this makes answerable
Four questions that are otherwise guesswork, and every one of them is a question somebody asks during an incident rather than in advance.
- Did it run? The most common question about background work, and the hardest to answer without a record. An absent row is an answer.
- How long does it take? Not on average across a system, but for this job on this queue — which is what tells you whether a nightly job is drifting towards its own interval.
- Is this failure new? The attempt count and the exception class together distinguish a transient network failure retrying from a job that has been failing the same way since a deployment.
- Which queue is backing up? Runs carry their queue, so a slow queue is identifiable rather than inferred from the symptom.
It is a record, not an archive
Job runs are kept for thirty days and then pruned, on the same nightly retention sweep that governs logs and sessions. That is the right window: long enough to investigate an incident and a previous month's comparison, short enough that a busy installation is not carrying years of rows describing jobs that succeeded.
They are also one of the handful of tables pruned once for the whole installation rather than per organisation, because a queue job belongs to the system rather than to any workspace. Pruned per organisation, rows carrying no organisation would match nothing and accumulate forever.
What sits on top
A queue health digest goes out each morning at half past nine, and a heartbeat records that a worker is alive every minute. Those two together answer the question the run records cannot: a job that never started leaves no row at all, so the absence of records is ambiguous between nothing to do and nothing running.
That is the general shape of monitoring background work. The runs tell you about work that happened. A heartbeat tells you the thing that does work is alive. Neither substitutes for the other, and a system with only the first is one where a dead worker looks like a quiet night.
Our take
The valuable decisions here are all defensive. Checking once whether the storage exists, decoding payloads cautiously, capping every stored string, and — above all — swallowing every failure into a log line rather than an exception. Each of those exists because the alternative turns a monitor into a cause of the incidents it was built to explain, and that failure mode is nastier than having no monitor: the jobs it breaks are the unusual ones, which are the ones you most needed to see. What the records make possible is unglamorous and constant — did it run, how long did it take, is this failure new, which queue is slow. Four questions that are pure guesswork without them and immediate with them, and all four get asked at the worst possible moment.
What AWRA OpsHub does today
- A record for every background job at start, at completion and at failure, across all queues and connections.
- Job name, queue, connection, attempt count, start and finish times, and runtime in milliseconds on every record.
- Exception class and the first thousand characters of the message on a failure, plus a link to the framework's own failed job entry.
- Runtime left empty rather than estimated when no matching start record exists, and a terminal record still created so the completion is not lost.
- A single cached check for whether the storage exists, so monitoring is a no-op before a migration rather than a failure.
- Every handler wrapped so a monitoring failure logs a warning and never propagates into the job being watched.
- Defensive payload reading, so a job on an unusual queue driver is recorded with less detail rather than failing.
- Every stored string length-capped before it is written.
- Thirty-day retention on the same nightly sweep as logs and sessions, pruned once for the installation rather than per organisation.
- A daily queue health digest and a queue worker heartbeat every minute.
More we can add to your workspace
- A runtime trend per job, so one drifting towards its own scheduled interval is visible before it starts overlapping.
- A workspace-visible view of the jobs that concern it, rather than records only the operator can read.
- Alerting on a failure rate rather than a single failure, so a transient error and a persistent one are treated differently.
- Correlation between a job run and what triggered it, so a failed export can be traced back to the request that asked for it.
- A queue depth history, which is what tells you a backlog is growing rather than merely present.
- Retention configurable per installation, for operators who want longer than thirty days.
Where we point you to a specialist
- We will keep the monitor incapable of failing a job. An observer that can break what it observes produces failures in exactly the unusual jobs it was most needed for, and losing an observation is always the better trade than losing the work.
- We will not estimate a duration for a job whose start was never recorded. An empty runtime is visibly unknown and a computed one is indistinguishable from a measurement, which is the wrong direction to be wrong in for a figure people use to decide whether something is getting slower.
- These records describe the platform's own execution rather than your business events. Where an operational fact matters to your organisation it belongs in your records with an audit trail, not inferred from whether a background job completed.
A runtime trend per job is the contained piece with the most operational value, because a nightly job creeping towards its own interval is invisible right up until the night it overlaps.
From records to warnings
The capture is thorough and safe. What turns it into operations rather than forensics is trend and alerting on top of records that already exist.
Runtime trends
How long each job takes over weeks, so one approaching its own interval is visible before the night it overlaps.
Rate-based alerting
A warning on a failure rate rather than on a single failure, so a transient error and a persistent one are treated differently.
Trigger correlation
A job run linked to whatever asked for it, so a failed export traces back to the person waiting for it.
We publish scope, not dates.
Scope operational monitoringFive questions to ask about background work
Is every job recorded?
A good answer sounds like
Yes, at start and end.
What ours actually is
Yes — a row at start, updated at completion or failure, with name, queue, connection, attempts, timings and runtime.
What happens if the monitor itself fails?
A good answer sounds like
The job survives.
What ours actually is
A warning is logged and the job proceeds. The observation is lost, which is the correct thing to lose.
Is a failure distinguishable from a retry?
A good answer sounds like
Yes.
What ours actually is
Yes — the attempt count and the exception class together separate a transient failure retrying from a persistent one.
How long are the records kept?
A good answer sounds like
A stated window.
What ours actually is
Thirty days, on the same nightly retention sweep as logs and sessions.
Would you know if the worker died?
A good answer sounds like
A heartbeat, not an absence.
What ours actually is
A heartbeat every minute, because no records is ambiguous between nothing to do and nothing running.
Check whether your background work is observable
The test is simple: can somebody answer "did that run, and how long did it take" without reading a log file. If not, every incident involving background work starts from zero.
Talk through monitoringFrequently asked questions
Does recording every job slow things down?
It is one insert at the start and one update at the end, against a table checked for existence once per worker instance rather than on every job. Set against the cost of a background job doing real work, it is negligible — and the alternative is an incident where nobody can establish whether the job ran at all.
What happens if the monitoring itself has a problem?
The job carries on. Every handler is wrapped so a monitoring failure produces a warning in the log rather than an exception, because a monitor that can fail the job it watches breaks exactly the unusual jobs it was most needed for. Losing the observation is always the better trade.
Why is runtime sometimes empty?
Because the completion could not find a matching start record, and estimating a duration from an assumed start would produce a number indistinguishable from a measured one. The terminal record is still created — the completion is never lost — the duration is simply reported as unknown, which is what it is.
How do we tell a one-off failure from a broken job?
The attempt count and the exception class together. A first attempt failing with a connection error and retrying is a different situation from the fifth attempt failing with the same exception since a deployment, and the two look identical in a bare failure count. Alerting on a rate rather than on individual failures is the piece that would automate the distinction.
How long are the records kept?
Thirty days, pruned on the same nightly retention sweep that governs logs and sessions — and pruned once for the whole installation rather than per organisation, since a queue job belongs to the system rather than to any workspace. Thirty days covers an investigation and a comparison with last month without accumulating years of successful runs.
Would this tell us if the queue worker had died?
Not on its own, and that is worth being clear about. A worker that is not running produces no records, and no records is ambiguous between nothing to do and nothing running. A separate heartbeat every minute resolves that, and the two together are what monitoring background work actually requires.
