A Backup Nobody Has Restored
Every backup job reports success, right up to the morning somebody needs one. The only test of a backup is a restore, and the two numbers everyone quotes about recovery are measurements you take, not promises you make. How we drill ours, what the drill checks, and what to demand of any vendor holding your records.
The finance system goes down on a Tuesday morning. Someone says the words everyone has been waiting years to say calmly: “It’s fine, we have backups.” And they do. The job has run every night for three years, the log says success every time, and the folder is full of files with tidy dates in their names. Then somebody tries to load last night’s file, and it stops two thirds of the way through, because the disk filled up during the dump eight months ago and every file since has been cut short at the same table.
The short version, first
A backup that nobody has restored is a hope with a timestamp. The job that writes it can only tell you that it wrote something; whether that something comes back as a working database is a separate question, and the only way to answer it is to restore it. So we restore ours on a schedule: a weekly drill loads the newest nightly dump into a separate scratch database, times the restore, checks that the core tables came back with data in them and that the table count matches the live system, and records the result. The two figures that matter, how long recovery takes and how much work it loses, come out of that record as measurements. This is how we operate the platform, not a setting in your workspace, and it is what you should ask any vendor to show you.
Why backups fail without telling anyone
A backup job has exactly one moment of truth, and it is not the moment it runs. Writing a file is easy to do and easy to report. What a backup job almost never does is read its own output back the way a restore would, so it cannot tell a complete dump from one that stops halfway, a dump of the right database from a dump of an empty one, or a file that will load in ten minutes from one that will take six hours. Each of those reports success.
- A truncated dump. The disk fills mid-write, the compressor closes the file cleanly, and the result is a valid archive of half a database.
- An upload that failed after the dump succeeded. The local file is fresh, so a check that looks for “a recent file” is green on precisely the night the offsite copy did not happen.
- A restore that works but takes too long. Nobody has timed it, so the recovery plan says “a couple of hours” because that sounded reasonable in a meeting.
- Credentials, tools or versions that drifted. The dump format changed with an upgrade, or the restore needs a privilege nobody has any more.
- The wrong thing, faithfully. A job pointed at a staging database, or at a database that was renamed, backing up nothing important with perfect reliability.
Every item on that list is invisible to a job that only writes. All of them show up the first time someone restores. The question is whether that first time is a scheduled drill on a quiet Sunday morning or the worst morning of the year.
The backup job can only tell you it wrote something. Whether it comes back is a different question, and a restore is the only thing that answers it.
Recovery time and recovery point are readings, not targets
Two acronyms dominate every conversation about disaster recovery. The recovery time objective is how long you can afford to be down. The recovery point objective is how much recent work you can afford to lose. Both are objectives, which is to say wishes, and a surprising number of contracts and policies state them without anyone having measured either.
The useful version of each is a reading taken from a real restore. Recovery time is the stopwatch on an actual load of an actual backup. Recovery point is the age of the backup you would be loading, at the moment you would be loading it. Neither can be estimated well from the size of a file or the schedule of a job, because the things that make them bad — slow disks, lock contention, a dump that grew threefold since anyone last looked — only appear when you do it.
How one drill turns into two figures (an illustrative Sunday)
Sixty minutes is the configured default and can be set per server. The 85-minute dump age is a property of the schedule, not of the system: a drill at 03:40 on a 02:15 dump will always read about that. What it tells you honestly is that the file restored was that old. What it does not tell you is worst-case loss on a bad day, which is the subject of a later section.
What the drill does, step by step
This is a platform control. It runs on the servers AWRA OpsHub operates, against the database every workspace lives in, and nobody in your organization has to configure it, schedule it or think about it. It is switched on per server once a scratch database exists for it; until then the scheduled command does nothing and the health screen says, in words, that restore drills are off. Here is what happens when it runs.
-
Find the newest nightly dump
The drill looks in the backup folder for the nightly dumps and picks the most recently written file. If there is none, that is itself a failed drill, recorded with the folder it looked in.
-
Refuse the live database
The target is a separate scratch database named in configuration. If it is unset the drill stops; if it names the live database the drill refuses outright rather than restore over production.
-
Clear the scratch database
Every table left in the scratch database from a previous run is dropped first, so the result reflects this dump and nothing older.
-
Decompress and load, under a stopwatch
The file is decompressed straight into the database client in one pipeline that fails if either half fails, so a corrupt archive and a SQL error are both caught. Credentials go through a temporary file readable only by the process, and the whole step has a timeout.
-
Count what came back
The drill lists the tables in the restored copy and counts rows in a set of named tables, then compares the table count with the live system.
-
Empty the scratch database again and record
The restored copy is dropped so customer data does not sit in a second place between drills, and the result is written as one row: file, size, when it was taken, restore seconds, dump age, every check and any error.
What counts as a pass
A drill passes only when every one of the following holds. Any one failing fails the whole drill, and the record says which, with the restored and live figures side by side.
| Check | Passes when | What it catches |
|---|---|---|
| The load itself | Decompression and import both exit cleanly within the timeout | A corrupt or truncated archive, a dump that will not load into the current database version |
| Core tables | Organizations, users, roles, permissions and the migration history all come back with rows in them | A dump of the wrong or empty database; a dump that has structure but no data |
| Table count | The restored copy holds at least 90% of the tables the live system has | A dump cut short partway, which is the full-disk failure from the opening |
| Recovery time | The restore took no longer than the configured objective, sixty minutes by default | A backup that restores, but too slowly to be the plan |
The drill also records row counts for items, customers, suppliers, purchase orders, invoices and the audit log, next to the live counts at the moment of the drill. Those are written down and never judged, on purpose. The live system has moved on since the dump was taken, so a restored invoice count lower than the live one is expected, and a rule that failed on it would fail every week and teach everyone to ignore it. The numbers are there so a person reading the record can see the restored copy was the right order of magnitude, not to trip an alarm.
Why ninety per cent, rather than every table
A dump taken at 02:15 can legitimately differ from the live schema at 03:40 if a release added a table in between. Demanding an exact match would make the check fail on a deploy rather than on a backup. Ninety per cent is loose enough to survive a new table and tight enough to catch a dump that stopped a third of the way through, which is the failure the check exists for.
Where a result goes
Every drill is a row in its own record, kept rather than overwritten, and the platform owner sees the full history on an owner-only page with the checks for each run shown as pills that turn red when they fail. The same page can queue a drill on demand, limited to three an hour because a restore is heavy work on a shared server.
The newest drill also feeds a card on the service health screen, the same screen we described in <a href="/blog/eleven-checks-and-one-verdict">Eleven Checks and One Verdict</a>. The card is green only when the last drill passed and is less than eight days old; a failure, a drill that has gone stale, or no drill at all turns it amber with a sentence saying which. The health screen’s checks go to the platform owners in a digest every morning. That is the alert path today: a failed Sunday drill is in the owners’ inbox on Monday morning, with the reason attached.
What the two numbers leave out
A drill is only as strong as the path it walks, and it is worth being exact about which path this one walks, because the same exactness is what you should expect from any vendor describing theirs.
It restores the local copy of the nightly dump. The offsite copy, the one you would actually need if the server itself were gone, is written by the same backup job and checked by its own health card, but the drill does not download it and restore from it. It restores the nightly tier only. Between nightly dumps, a second tier writes a full dump on a short interval to hold down worst-case loss; the drill does not restore those, so the recovery point it records is the nightly file’s age, not the intraday one’s. And it restores onto the same database server as the live system, so its stopwatch measures a busy production machine rather than a fresh one being stood up in an emergency, which could be faster or slower.
What each kind of evidence actually proves
| Question a restore must answer | “The backup job succeeded” | This weekly drill |
|---|---|---|
| A file was written on schedule | Yes | Yes |
| The file decompresses and loads end to end | No | Yes |
| The core data is present, not just the structure | No | Yes |
| The dump is complete, not cut short | No | Yes |
| Recovery time is a measured figure | No | Yes |
| The offsite copy can be fetched and restored | No | No |
| A fresh machine can be stood up from it | No | Partly — configurable by you |
| The application reads the restored records correctly | No | Partly — configurable by you |
Built and maintained Configurable by you, not maintained by us Not built
Part means the drill covers some of the question: it proves the database loads onto a database server, which is the bulk of standing up a new machine but not all of it, and it counts rows without reading them through the application.
We would rather say that plainly than let the word “tested” cover all eight rows. Most of the value is in the first five: the failures that actually hurt organizations are truncated dumps, empty dumps and restores nobody ever timed, and those are the ones a weekly drill catches on a Sunday instead of on the day it matters.
What to demand of any vendor holding your records
You cannot run a restore drill on a vendor’s infrastructure, but you can ask questions that tell a drilled backup from a hoped-for one. We set out the wider set of hosting questions in <a href="/blog/ten-questions-about-where-your-data-lives">Ten Questions to Ask Any ERP Vendor About Where Your Data Lives</a>; these are the ones specific to recovery.
Before you accept “we have backups”
- The date of the last restore, not the last backup, and what was restored.
- The measured restore time from that drill, and whether it was inside the stated objective.
- What the drill checked after loading, in concrete terms: tables, row counts, a checksum, an application read.
- Whether the drill restores the offsite copy or a local one.
- Who learns that a drill failed, and how quickly.
- Whether the stated recovery point is the backup interval or a figure measured from a real drill.
Walking more of the path, on purpose
Each of these extends the drill one step further along the route a real recovery would take. They are platform work rather than workspace features, and organizations with a contractual recovery requirement are the ones who would ask for them first.
Restore from the offsite copy
Fetch the newest dump from offsite storage and restore that, so the drill proves the copy you would need if the server were lost, including the time it takes to download.
Drill on a second machine
Run the restore on a separate server, so the stopwatch measures standing up a replacement rather than loading onto the live database server.
Drill the intraday tier too
Restore the newest short-interval dump as well as the nightly one, so the recorded recovery point reflects the tightest copy the platform actually holds.
An immediate alert on a failed drill
Raise a failed drill the moment it fails, through the same channel the backup job already uses for its own failures, rather than in the next morning’s digest.
The first and fourth are the cheapest and close the two questions buyers ask most: is the offsite copy good, and who knows when it is not.
Tell us what your recovery terms requireWhat AWRA OpsHub does today
- A weekly restore drill on the platform, scheduled for Sunday at 03:40 and switched on per server once a scratch database is configured.
- A refusal to restore over the live database, by name, before anything is loaded.
- A decompress-and-load step that fails on either half, with a timeout, so a corrupt archive and a SQL error are both caught.
- Pass rules for core data, table completeness and recovery time, each recorded with the restored and live figures side by side.
- Measured recovery time and dump age on every drill, kept as a history rather than overwritten.
- An owner page with the drill history and an on-demand run, rate-limited because a restore is heavy work.
- A service health card that turns amber on a failed drill, a drill older than eight days, or drills switched off, carried into the owners’ daily digest.
More we can add to your workspace
- A drill that restores the offsite copy, fetched from offsite storage, so the copy needed after losing the server is the one proven.
- A drill on a separate machine, so recovery time is measured on a fresh server rather than on the live database server.
- A drill of the intraday dumps, so the recorded recovery point matches the tightest backup the platform keeps.
- A checksum of each restored table, compared against the same tables at the moment the dump was taken.
- An application-level read of the restored copy, opening a sample of records through the product rather than counting rows.
- An immediate alert on a failed drill, through the channel the backup job already uses, in place of the next morning’s digest.
- A drill summary on the trust centre, showing the date and result of the latest restore to buyers doing due diligence.
Where we point you to a specialist
- We do not quote a recovery time we have not measured. If a contract asks for one, the figure we give comes from a drill record, with its date, and we will say which path that drill walked.
- We do not restore into the live database to test a backup, and the drill refuses to by name. A test that can damage what it is testing is not a test anyone should be allowed to schedule.
- We will not treat a successful backup job as evidence that the backup restores. The job’s status and the drill’s result are reported separately on purpose, and only the second answers the question a recovery depends on.
The offsite restore and the immediate alert are the natural first pair: one proves the copy that matters most, the other makes sure a bad week is noticed within minutes. The second machine and the intraday drill follow once the offsite path is in place, and the trust centre summary is a small addition once there is a drill history worth publishing.
Four questions that separate a drilled backup from a hopeful one
When did you last restore a production backup, and what did you restore it into?
What you will probably hear
We back up every night and keep thirty days.
How to read it
That answers a different question. A schedule and a retention period describe writing. Ask again for a date and a target; if nobody can name either, the backups are untested, however many there are.
How long did that restore take?
What you will probably hear
Our recovery time objective is four hours.
How to read it
An objective is a wish. Ask for the stopwatch reading from the last drill and whether it was taken on hardware like the hardware you would recover onto.
What did you check after the restore finished?
What you will probably hear
It completed without errors.
How to read it
An import with no errors can still be an import of half a database. Look for concrete checks: core tables present with data, a table count against live, ideally row counts or checksums and a read through the application.
If this week’s drill failed, who would know, and when?
What you will probably hear
Our team monitors backups.
How to read it
Ask for the mechanism: a record, an alert, a person. A drill whose failure goes nowhere is a log file, and a log file nobody reads is the same as no drill.
Ask us for the drill, not the brochure
If your organization is weighing where its records should live, ask us what the latest restore showed and which path it walked, and hold every other vendor to the same question. The wider picture of how we host and protect workspaces is on the <a href="/trust-center">trust centre</a>.
Talk to us about recoveryFrequently asked questions
What is a restore drill?
A restore drill loads a real backup into a separate database on a schedule, times how long it takes, and checks that what came back is complete. It is the only way to know a backup works, because the job that writes a backup cannot tell a complete file from one that was cut short.
Do I need to set up restore drills in my AWRA OpsHub workspace?
No. The drill is a platform control that runs on the infrastructure AWRA OpsHub operates, covering the database every workspace lives in. There is nothing to configure in your workspace.
What is the difference between recovery time and recovery point?
Recovery time is how long it takes to get a working system back from a backup. Recovery point is how old that backup is, which is how much recent work would be lost. Both are most useful as figures measured from a real restore rather than as targets written into a policy.
What makes a drill fail?
A drill fails if no backup file is found, if the file will not decompress or load, if any core table comes back empty, if the restored copy has fewer than ninety per cent of the live system’s tables, or if the restore takes longer than the configured recovery time objective, which defaults to sixty minutes.
Does the drill ever touch the live database?
No. It restores into a separate scratch database and refuses to run if that setting names the live database. The scratch copy is emptied again after each drill so customer data does not sit in a second place between runs.
What should I ask a software vendor about their backups?
Ask for the date of the last restore rather than the last backup, the measured restore time, what was checked after the restore, whether the offsite copy was the one restored, and who is told when a drill fails.
