A Timestamp That Proves Nothing
A field called "notified at" was being filled in on tickets where nobody had been notified. The absence of an alert is a gap you can find. A false record of one is a gap that closes the investigation.
There is a category of software defect that is worse than the thing it is hiding, and it has a recognisable shape: the system does something, fails, and then writes down that it succeeded. Nobody is lying. The failure and the record are in different lines of code and the second one was never told about the first. But the result is a database that will confidently answer a question wrongly, forever, and be believed — because a timestamp is the most trusted thing in any system and almost nobody asks what wrote it.
The specific one we found
A scheduled job sweeps for tickets that have gone past their deadline, tells somebody, and marks the ticket so that the next sweep does not tell them again. Marking is necessary; without it a breached ticket generates an identical warning every few minutes until somebody either fixes it or turns the alerts off, and they always turn the alerts off.
The marking happened whether or not anybody had been told. The step that sent the message and the step that recorded it as sent ran one after the other, unconditionally, and the second one never asked the first how it went.
One breached ticket with nobody to notify
Follow the fourth row, because it is where a reporting inaccuracy becomes an operational one. The marker was the same field the sweep used to decide what to look at. So writing it did not merely produce a wrong record — it removed the ticket from every future run. The one breach nobody had been told about became the one breach that could never be raised again. A missed alert became a permanently missed alert by the act of recording that it had not been missed.
Why this is worse than sending nothing
Consider what each version does to somebody investigating afterwards. A ticket sat for three days past its deadline and a customer is unhappy, and somebody sensible asks the obvious question: was anybody told?
- No record at all. The answer is "we don't know", which is unsatisfying and correct, and it sends the investigation somewhere useful — usually to whether the alerting works at all.
- An accurate record of failure. The answer is "no, and here is why". Best case. The system knew, and said so.
- A record of success that did not happen. The answer is "yes, at 19:45 on the twelfth". The investigation stops. The conclusion is that a human ignored the warning, and the next conversation is about that person rather than about the system that never warned them.
What the third one costs, exactly
It moves the fault from the software to a colleague, with a timestamp as the evidence. Nobody set out to do that and it is what the record does. A system that cannot tell you it failed will eventually let somebody be blamed for its failure, and it will do so in writing.
This is the whole argument for treating an evidence field as a different kind of thing from an ordinary field. A quantity that is wrong gets noticed, because quantities are compared to other quantities. A timestamp asserting that a thing happened is compared to nothing. It is read once, years later, by somebody with a question, and it answers.
What it does now
The step that sends the alert reports how many people it actually reached, and the ticket is marked only when that number is not zero. A breach the system could not deliver stays unmarked, which means it is examined again on the next sweep, and the one after — so the alert finally goes out on the day somebody is assigned to the ticket or a colleague joins the department it was queued in.
And the sweep no longer finishes quietly when it could not deliver. It reports the count of breached tickets it had nobody to tell about, because that number is itself the thing worth knowing: it is not a technical failure, it is a queue with no owner, and somebody should hear about it in that form.
The question this leaves for every system you own
This defect took an afternoon to fix and its class is everywhere. Anywhere a system records that it did something outward-facing — sent, delivered, transmitted, filed, notified, escalated — there is a place where the record and the act can come apart, and the record is the half that survives.
- Does this field record an attempt or an outcome? They are different, they are usually named the same, and the name almost always implies the second one.
- Who wrote it, and did that code know whether the thing worked? A record written next to an action rather than by it is a record of intent.
- Is this field also a filter? If the system uses it to decide what to process, an incorrect value does not just misinform — it changes behaviour, and it hides the case it was wrong about.
The third question is the one worth stealing. It is what turned this from an untidy record into a breach that could never be raised again, and it applies to every "already handled" flag in every system: the moment such a flag is written optimistically, the population it is wrong about is exactly the population that becomes invisible. Ours was wrong for a year and is right since August 2026, and the table above is from our own test suite rather than from a description of one.
