The Audit Trail That Leaves the Building

At 02:14 on a Saturday somebody signs in with an administrator’s credentials, changes a supplier’s bank account, approves a payment run and exports the customer list. At 02:31 they reach the database itself, delete the last forty audit rows, and leave. On Monday the finance lead notices a payment nobody remembers approving, opens the audit log, and finds a perfectly ordinary Friday afternoon followed by nothing at all. The log is intact. Every check on it passes. It simply ends early.

That is the failure this post is about, and it is not the one most audit-trail discussions worry about. The usual fear is an edit: someone changes a row to say a different person approved the payment. Good audit design catches edits. The harder case is the trail that has been shortened — the most recent entries removed, so that what remains is internally consistent, honest about everything it still contains, and silent about the only hour that mattered.

No amount of cleverness inside the system fixes that, because the system is what was compromised. The fix is boring and physical: a copy of every event, made within a minute of it happening, held somewhere the attacker would also have to break into. This is what security teams mean when they ask whether you can “stream to our SIEM”, and it is worth understanding why they ask before deciding whether the answer you are given is any good.

Why a chain alone cannot see the end of itself

Every audit row in AWRA is signed over its own content and the signature of the row before it, so an edited row stops matching its signature and a deleted row leaves its successor pointing at nothing. We wrote about that mechanism, and about how a legitimate retention purge is explained inside the same chain, in The Excuse Is Also in the Chain. It is good at what it does. Here is what it cannot do, and no design of its kind can.

• It cannot see a missing tail. Each row vouches for the one before it. Nothing vouches for the row after the last one, so removing the newest rows leaves a shorter chain that verifies cleanly.
• It accepts a start above one. Retention removes the oldest entries on purpose, so a chain beginning at entry 4,812 is normal and is treated as an anchor rather than as damage.
• It is checked by the system it protects. The verifier runs on the same servers and reads the same database. Whoever owns both can also decide what the verifier gets to read.
• Its signing key lives with the application. The key is deliberately not in the database, so a database leak alone cannot re-sign rows. An attacker who holds the application server as well could, in principle, re-sign a rewritten history from top to bottom.

None of those is a flaw in the chain. They are what “self-contained” means. A tamper-evident log tells you that something changed relative to a copy you trust, and inside the building there is no copy the intruder could not also reach. The chain supplies the fingerprints; the external store supplies the second set of records to compare them against.

An audit trail stored only inside the system it audits is a statement the system makes about itself. Evidence is what somebody else kept.

What leaving the building actually buys

Go back to Saturday night with a stream switched on. At 02:14 the sign-in is written to the audit log; within a minute it is in the security team’s log store, alongside the bank-detail change at 02:16 and the approval at 02:19. When the intruder deletes the last forty rows at 02:31, they are deleting copies. The originals left the building fifteen minutes earlier.

On Monday the comparison is mechanical rather than forensic. The log store holds sequence numbers up to, say, 10,447. The application’s own log ends at 10,407. Each event in the store carries the signature of the event before it, so the store can show that its own copy is unbroken from 10,400 to 10,447 — and therefore that the forty missing rows existed, in that order, with those signatures, before anyone removed them. The question stops being “did something happen?” and becomes “what do these forty rows say?”, which the store can already answer.

The last line is the quiet benefit. A SIEM is where the sign-in from an unfamiliar country, the identity provider’s risk alert and the bank-detail change can sit on one timeline. An audit log that only lives inside the operations system can never be joined to the rest of the evidence without someone exporting it by hand — usually after the incident, which is exactly when the export is least trustworthy.

How the stream works

Streaming lives under Settings → Audit Log Streaming. You choose a format, paste an HTTPS endpoint and a token, and switch it on. Once a minute a scheduled job looks at every workspace with streaming on and sends whatever is new.

1. Read from a cursor

   Each stream remembers the last audit entry it delivered. A run reads the next entries after that point, oldest first, five hundred at a time — so a backlog after an outage drains in order rather than in whatever order the database prefers.

2. Send the batch

   The batch goes out in the destination’s own shape: newline-delimited events with the Splunk token header, an array of log entries with the Datadog API key, or a JSON body carrying a Bearer token and an HMAC-SHA256 signature of the body in X-Awra-Signature so your receiver can confirm it came from us unaltered.

3. Move the cursor only on acceptance

   The cursor advances only when the destination answers with a success status. A timeout, a refused connection or an error status leaves it exactly where it was, so the same entries are offered again on the next attempt and nothing is skipped.

4. Back off, then try again

   A failed send waits one minute, then two, four, eight, sixteen, thirty-two, and then an hour between attempts until the destination recovers. The first success clears the error and resets the wait.

One honest consequence of “move only on acceptance”: delivery is at least once, not exactly once. If your endpoint accepts a batch and the answer is lost on the way back, we cannot tell that from a failure, so the batch is sent again. Every event carries its audit id and its chain sequence number, which makes de-duplication on the receiving side a one-line search. We chose a possible duplicate over a possible gap, because a duplicate is visible and a gap is the thing this whole feature exists to rule out.

What an event carries

Every event is one audit entry, in full, plus its place in the chain. The receiving side gets enough to answer who, what, when, from where and to which record without coming back to us.

Notice what the receiving side can and cannot do with the last row. With the sequence numbers and the two signatures it can prove its own copy is continuous — every event links to the one before it, and no number is skipped. What it cannot do is recompute a signature from the content, because the signing key stays with the application. That is deliberate: a key shared with every customer’s log platform would be a key anyone could sign with. The stream’s job is to put an untouchable copy of the signatures somewhere else; the comparison is between that copy and what the application says today.

The event is not minimised. The actor’s name, email address and IP address leave with it, because an investigation that has to come back to us to find out who “user 41” was is an investigation that depends on the system under suspicion. The consequence is worth saying plainly: once an event is in your log store, your log store’s retention governs it. When AWRA’s own retention policy later anonymises an old entry, the copy you already hold is untouched. That is the point of holding it, and it means the SIEM’s retention needs to match your privacy policy as well as your security one.

Who controls the stream, and how you know it is working

Anyone with permission to view settings can see the page; only someone who can edit settings can change the destination, switch it off or send a test. The token is stored encrypted, never shown again after it is saved, and left blank on later edits to keep the current one. The API status view reports the host, format, last success, last error and backlog, and omits the token entirely.

Every save is itself an audit entry naming the destination host and the format, though never the token. Pointing the stream at a different host therefore shows up — in the application’s log, and in the stream, which delivers that entry to the new destination. Since an attacker’s first move against a stream is to redirect it or switch it off, your log store should also alert on silence: a workspace that normally produces events every few minutes and has produced none for an hour is a finding, whatever the reason.

The settings page shows three numbers — whether streaming is on, when the last batch was delivered, and how many entries are waiting — and, after a failure, the destination’s own answer and the time of the next retry. Send a test event pushes one synthetic entry through exactly the same path, so a wrong token or a firewall rule fails on the settings page rather than quietly at 02:14 on a Saturday.

What streaming does not promise

A stream shrinks the window in which a trail can be quietly shortened; it does not close it to zero. An entry written at 02:14 and deleted from the database at 02:14:30, before the next send, never left the building. While your endpoint is down, everything in the backlog is in the same position. The protection is proportional to how promptly events leave and how reliably your destination accepts them, which is why the waiting count and the last-error line are on the settings page rather than in a log file.

Nor does streaming replace the permanence inside the application. Audit entries cannot be deleted from any screen — the argument for that is in An Audit Trail Nobody Can Delete — and the chain still catches edits and gaps between rows. The three work as layers: nobody can remove an entry through the product, nobody can alter one through the database without the chain noticing, and nobody can shorten the trail without the external copy disagreeing.

Frequently asked questions