Numbers First, Words Second: How the AI Writes About Your Data
The paragraph at the top of an AI dashboard is the part everyone reads and the part nobody interrogates. Here is the architecture underneath ours: every figure computed in plain code, the model given the figures and told not to invent any, and a fingerprint that throws the paragraph away the moment the numbers move.
There are two ways to build the sentence at the top of an analytics screen that says "Your backlog grew 14% this month and three tickets have breached — clear the breaches first." One is to hand your data to a language model and ask it what it thinks. The other is to compute every number yourself, hand the model the finished numbers, and ask it only to put them in a sentence.
They look identical on screen. They are completely different products. The first can be confidently wrong about arithmetic, which is the one thing a model has no business being trusted with; the second cannot, because it never did the arithmetic. We built the second, and this article is about how the seam works — including the two numbers on those cards that are not what their labels suggest, and the bug we found in our own prompt while writing it.
The seam, in five steps
-
Your database is queried, in our code
A module's insight builder runs ordinary queries against your data: open tickets, breaches, unassigned, resolved this month, average satisfaction. Counts and sums, in PHP, with no model anywhere near them.
-
A score and a summary line are assembled
Still arithmetic. A health score out of 100 built by subtracting penalties, a one-line summary like 14 open · 3 SLA breaches · CSAT 4.2/5, a set of chips, any alerts, and one recommendation chosen by a short chain of if-statements.
-
The whole insight is flattened into labelled facts
The area name, the summary line, every metric as label: value, whatever bullet lines the module publishes, and the current recommendation sentence — capped at thirty lines so a prompt cannot bloat.
-
A short prompt asks for prose and forbids invention
Two to four sentences, plain language, for a busy manager, naming the single most important next action, using only the facts supplied and inventing no figures. Plain text, no markdown, no bullets.
-
The paragraph is cached against a fingerprint of those facts
Not just a time-to-live. A short hash of the flattened facts is stored with the summary, so when the underlying figures change the cached paragraph is treated as stale immediately — even if it was generated four minutes ago.
That last step is the one that matters most for trust, and it is the one most implementations get wrong. A cached narrative is dangerous in a way a cached number is not: numbers on a page carry an implicit "as at", and prose does not. A paragraph saying "nothing is breaching" sitting above a card showing three breaches is worse than no paragraph at all, because a manager reads the sentence and stops.
A stale number looks stale. A stale sentence looks authoritative.
Why the narrative is fingerprinted, not just timed
What is arithmetic and what is the model
It is worth being exact about which parts of an AI screen would still be there with every provider switched off, because on our screens most of it would.
Where each thing on an AI insight card comes from
| Element on the card | Computed in our code | Written by the model | Survives with AI off |
|---|---|---|---|
| Every metric — counts, sums, rates, balances | Yes | No | Yes |
| The one-line summary strip | Yes | No | Yes |
| The health score out of 100 | Yes | No | Yes |
| The confidence figure | Yes | No | Yes |
| Alerts, chips and the recommended action | Yes | No | Yes |
| The demand forecast and its risk bands | Yes | No | Yes |
| The anomaly and pricing radars | Yes | No | Yes |
| The support queue risk ranking and its reasons | Yes | No | Yes |
| The two-to-four sentence narrative paragraph | No | Yes | No |
Built and maintained Configurable by you, not maintained by us Not built
One row out of nine depends on a language model. That ratio is the architecture, and it is why we are comfortable telling customers exactly what leaves the server — because the answer for eight of these rows is nothing at all.
The bug we found writing this article
The prompt was missing the numbers it was asked to summarise
The flattening step accepted metrics in a rich form — a list of label-and-value objects — and every insight builder in the codebase actually emits the plain form, a flat map of name to number. The result: the metric loop silently skipped all of them, and the model was asked to "summarise these computed insights" using only the area name, the one-line summary and the recommendation sentence. For the six modules with no bullet lines, that meant the narrative was a paraphrase of a summary strip rather than a reading of the figures. It has been fixed — both shapes are now flattened, null values render as n/a, and the fingerprint changes as a result, so every cached narrative regenerates once against the fuller facts.
Two things are worth drawing out of that, because they generalise beyond our codebase.
First: the output was plausible throughout. Nobody spotted it from reading the narratives, because a summary strip contains real numbers and a paragraph built from it reads exactly like a paragraph built from more. That is the characteristic failure of a language-model feature — it degrades into something that still sounds right. A broken query throws an error; a thin prompt just gets vaguer.
Second: the constraint held. The prompt says use only the data below and invent no figures, and the narratives did not hallucinate the missing metrics — they wrote less. That is the behaviour you want from a badly-fed prompt, and it is the argument for writing the instruction that way even when you are confident the facts are complete.
The two numbers that are not what they sound like
Every insight card carries a score and a confidence, both out of 100, both computed by hand-written arithmetic. Neither is a statistical quantity, and the second one is genuinely misnamed.
The score is a penalty model, and it is honest about it if you know how to read it. A module starts at 100 and loses points for the things you would deduct for yourself. On the support desk: up to 40 points off for the share of open tickets that are breaching, up to 20 off for unassigned tickets at 2 points each, and 10 points for every full point that satisfaction sits below 4 out of 5. There is no learning, no model, no training data. It is a rubric somebody wrote down, and its value is that it moves in the right direction for the right reason.
The confidence figure is a different matter. On the support desk it is 30, plus 2 points for every open ticket up to a ceiling of 45, plus a point for each ticket resolved this month up to 15, plus 10 if any satisfaction rating exists at all. Read that again: confidence rises with volume. A desk with 23 open tickets is at the maximum on the largest term, whether those tickets are well-managed or a shambles. It is a data-sufficiency indicator — is there enough here for the score to mean anything — wearing a word that implies a probability.
Two support desks, same confidence, opposite realities
Illustrative, and the arithmetic is exactly what runs. Confidence is answering "do I have enough tickets to say anything?" and both desks do. Read the score for how you are doing and the confidence as a completeness check on the score — and if confidence is low, the honest interpretation is not "the AI is unsure", it is "there is not much data here yet".
One more caveat on those two numbers, and it is the sort of thing vendors leave out. If a module's queries fail — a missing table on a partial installation, a permissions problem — the card falls back to a starter state: score 50, confidence 15, and an encouraging summary about insights appearing as data arrives. Those two values are indistinguishable from a genuine mid-range reading unless you notice the wording. If a card ever reads exactly 50 and 15, treat it as an error rather than a diagnosis.
How much to trust each layer
It helps to hold these in a deliberate order rather than treating everything on the screen as equally solid.
Trust, from strongest to weakest
The metrics
Counts and sums straight from your data. If the underlying records are right, these are right. Act on them.
The recommended action
A short if-chain: breaches first, then unassigned, then due-soon, then "healthy". Not clever, entirely predictable, and almost always the right priority.
The health score
A rubric somebody wrote. Trust its direction and its movement over time far more than its absolute value. A score falling from 82 to 61 is a real signal; 61 itself is a convention.
The narrative paragraph
Constrained to the supplied facts and forbidden to invent figures, but still generated prose. Useful for orienting someone in ten seconds. Never the basis for a decision on its own.
The confidence figure
A volume proxy, not a probability. Read as "is there enough data here", and remember that 15 exactly means something went wrong.
The practical rule that falls out of this: decide from the metrics, orient from the narrative. If a paragraph tells you something the numbers on the same card do not support, the numbers win and we want to hear about it.
Why the paragraph arrives late
You will notice that a narrative is often absent the first time you open a page and present a moment later. That is deliberate and worth understanding rather than reporting as a fault.
A call to a language model takes seconds and can fail. Blocking a page render on it would make every dashboard as slow as the slowest provider and as reliable as the least reliable one. So the request handler checks the cache, and if there is nothing valid it queues a background job and returns the page immediately with the numbers — which is the part you needed. The job generates the paragraph, stores it with its fingerprint, and it appears on your next load.
Two consequences follow honestly. A deployment with no queue worker running will never show a narrative at all, while every number continues to work perfectly — which is a confusing symptom with a mundane cause, and the first thing to check if paragraphs never appear. And a provider outage produces no error and no retry storm: the job returns without caching, the next page view tries again, and the screen simply carries on without prose.
What we do and do not do
What AWRA OpsHub does today
- Every figure on every AI card computed in our own code against your database — no model produces a number anywhere in the product.
- A prompt that supplies the computed facts and explicitly forbids inventing figures, capped at thirty lines so it cannot bloat with data volume.
- Fingerprint-based cache invalidation: a hash of the facts is stored with the paragraph, so changed numbers make the paragraph stale immediately rather than at the end of a six-hour window.
- Background generation on a queue, so a page never blocks on a provider and an outage degrades to no paragraph rather than a slow or broken screen.
- A manual refresh on the AI hub for the impatient, which clears the cached paragraph and the last-refreshed stamp.
- A fully transparent risk ranking on the support hub: arithmetic you can read, with the specific reasons — breached, due soon, first response overdue, high priority, reopened, unassigned — printed next to each ticket.
- Fail-soft throughout: no provider, a timeout, or unparseable output all produce silence rather than an error.
What it does not do
- Confidence is a volume proxy, not a probability, and the label does not say so. We would rather tell you here than let you infer a statistical guarantee that does not exist.
- A failed insight card reads score 50, confidence 15, which is indistinguishable from a genuine mid-range result except by the wording of its summary.
- No narrative history. Each paragraph replaces the last, so you cannot look back at what the summary said in March or diff it against today.
- No way to tune the narrative's length, tone, language or reading level, and no Swahili option.
- No per-module switch for narratives — they follow the deployment-wide provider setting like every other AI feature.
- The health-score rubrics are not configurable and not published in the interface. The weights are in our code, not in a settings screen you can adjust to your own priorities.
- No alerting on a narrative. It is text on a page you visit; nothing pushes a summary to you on a schedule.
The two we would fix first are cosmetic-sounding and are not: renaming confidence to something honest like data sufficiency, and making a failed card look failed. Both are about not letting a label imply more than the arithmetic delivers, which is the whole point of building it this way round.
Our take
Ask any vendor selling you an AI dashboard one question: which numbers on this screen were computed by your code, and which by the model? If the answer is that the model produces figures, you are being asked to trust prose about arithmetic to a system that is good at prose and structurally indifferent to arithmetic. Our answer is that every number here is ours and one paragraph is the model's, which is why switching the provider off costs you a paragraph and nothing else. Read the metrics to decide, read the narrative to orient, read the score for its direction rather than its value, and read confidence as a data-sufficiency note. That ordering will serve you on any analytics product, not only this one.
Read alongside this: what leaves your server gives the exact payload of each request, what an in-app AI assistant can and cannot answer covers the site assistant, the anomaly radar covers the rule-based alerting these cards sit beside, and forecasting demand without pretending to predict the future is the same computed-first philosophy applied to stock.
Numbers you can trace, prose you can ignore
Every score, count and forecast computed in our own code. The narrative is a paragraph on top, fingerprinted against the figures, and the dashboard works completely without it.
See Intelligence InsightsFrequently asked questions
Does the AI calculate the numbers on the dashboard?
No. Every metric, score, rate, forecast and alert on every AI card is computed by our own code running ordinary queries against your database. The language model receives the finished figures and is asked for two to four sentences of prose, with an explicit instruction to use only what it was given and invent no figures. This is the single most important architectural fact about the feature: switch every provider off and the numbers are unchanged, because they were never the model's to produce.
Why is the narrative sometimes missing when I open the page?
Because it is generated on a queue rather than during the page request. A model call takes seconds and can fail, and blocking the dashboard on it would make every screen as slow as the slowest provider. So the page checks the cache, queues a job if there is nothing valid, and returns immediately with the numbers. The paragraph appears on a later load. If narratives never appear at all while the numbers work perfectly, the usual cause is mundane: no queue worker is running on the deployment.
Can the narrative contradict the numbers above it?
It should not, and the design goes to some trouble to prevent it. The paragraph is stored with a short fingerprint of the exact facts it was written from, and it is discarded the moment those facts change — so a summary describing last week's position cannot sit above this morning's figures even inside its cache window. If you ever do see a paragraph that disagrees with the card it sits on, the numbers are authoritative and we would genuinely like to see the screenshot.
What does the confidence figure actually mean?
It is a data-sufficiency indicator, not a probability, and the label oversells it. On the support desk it is a base of 30 plus two points per open ticket up to a ceiling, plus a point per ticket resolved this month up to a smaller ceiling, plus ten if any satisfaction rating exists. It therefore rises with volume, not with quality — a chaotic desk and a well-run desk with the same number of tickets score the same confidence. Read it as "is there enough data here for the score to mean anything", and read the score itself for how you are doing.
A card shows score 50 and confidence 15. Is that bad?
It is probably not a reading at all. Those two exact values are the starter state a card falls back to when its queries fail — a missing table on a partial installation, a permissions problem, a module not set up. The summary text will say something about insights appearing as data arrives rather than reporting a position. This is a genuine weakness of the current design, because a failure looks like a mediocre result. Treat exactly-50-and-15 as an error to investigate rather than a diagnosis to act on.
Can we change the tone or language of the narrative?
Not today. The prompt is fixed — two to four sentences, plain language, aimed at a busy manager, one recommended action — and there is no setting for length, tone, reading level or language, so no Swahili option. The paragraph is also not stored historically: each one replaces the last, so you cannot compare what the summary said last quarter with today. If a configurable prompt or a bilingual narrative would matter to your organization, both are contained pieces of work and worth telling us about.
How is the support hub's "focus" list ordered — is that AI?
No, and it is our favourite example of why arithmetic beats inference for this kind of job. Each open ticket earns points: a hundred for a breached resolution target or forty for one due within four hours, thirty for an overdue first response, forty down to zero for priority, ten per previous reopen, fifteen for being unassigned, and one per twelve hours of age up to twenty — then the whole score is divided by three if the ticket is waiting on the requester, because that clock is paused. The top fifteen are shown with their reasons printed next to them. It is completely transparent, it is identical every time for the same inputs, and you can argue with the weights, which you cannot do with a model.
