Almost every company says it has backups. Far fewer have restored one. The difference matters on the day a server is lost, a database is corrupted or someone runs a delete on the wrong system. That is a poor moment to learn that the backup job stopped four months ago, or that the files are complete and nobody knows how to bring the system back from them.
How backups fail
- The job stopped and nobody noticed. A changed password, a full disk or a server move ends the schedule silently.
- The backup is stored on the same server. When the server is lost, so is the backup.
- The file is incomplete. The dump was cut off part way and is too small to contain the data.
- Only part of the system is covered. The database is backed up, the uploaded files and the configuration are not.
- The restore needs something that is gone. An encryption key, a specific software version, or a person who has left.
None of these is visible until a restore is attempted.
Set two targets
Two questions define what you need. How much data can you afford to lose? If the answer is a day, nightly backups are enough. If it is minutes, you need continuous archiving. This is the recovery point objective.
How long can the system be down? If the answer is an hour, you need a rehearsed procedure and probably a standby. If a day is acceptable, a simpler setup will do. This is the recovery time objective. Decide both with the business, since they determine the cost.
What to back up
List everything needed to rebuild the service from nothing.
- Databases.
- Uploaded files and documents.
- Configuration and environment settings.
- Secrets and keys, stored separately and securely.
- The definition of the infrastructure itself: which servers, which settings, which DNS records.
Application code is normally safe in version control. The items above usually are not.
Where to keep it
A long-standing rule is three copies of the data, on two different kinds of storage, with one copy in a different location. For a small system this can be as simple as the live database, a nightly dump on the server and a copy of that dump in cloud storage with a different provider.
Protect the copies from deletion. If an attacker or a mistake can reach the backups with the same credentials as the live system, they will be lost together. Use storage that does not allow backups to be changed or removed for a set period.
Test the restore
A restore test has two levels. The first is automatic and frequent: load the latest backup into an empty database and compare a few counts with the live system. This proves the file is complete and usable. It can run every night without anyone watching.
The second is a full drill, done a few times a year: rebuild the whole service on fresh infrastructure using only the backups and the written instructions. Time it. The duration is your real recovery time, and it is almost always longer than the estimate.
Write the procedure down
The steps to restore should be a document that someone other than its author can follow, under pressure, at an awkward hour. Have a different person run the drill each time. Every point where they have to ask a question is a gap in the document.
Monitor the backup job
Alert when a backup fails, and also when no backup has completed within the expected time. Check the size of each file against the previous one. A backup that is suddenly a tenth of its usual size has gone wrong even if the job reported success.
Snapshots are not a complete answer
Server snapshots from a hosting provider are useful and have limits. They sit with the same provider and account as the server. They capture the disk at one moment, which may leave a busy database in an inconsistent state. Keep them, and keep proper database backups elsewhere as well.
Summary
Agree how much loss and downtime are acceptable, back up everything needed to rebuild, keep a copy off the server and out of reach of the live credentials, test restores automatically and by drill, and write the procedure down. Backup and recovery design is part of our cloud and DevOps service. If you are not certain your backups would work, let us test them with you.