N100FN100F
Published on

Designing for failure

Everything fails sooner or later. A mature architecture decides in advance how long it can be down and how much data it can lose, designs to meet that and tests it before needing it.

Service line Cloud Architecture

The question is not whether it fails

Disks, regions, providers, deployments, people: everything fails at some point. An architecture is not measured by whether it fails, but by how much damage it does when it fails and how long it takes to come back.

That is not improvised during the incident. It is decided beforehand, designed and tested.

Two numbers that decide everything

Timeline of an incidentAn illustrative incident: the data-loss objective is met and the recovery-time objective is not.ARCHITECTURE · RECOVERYWhat was agreed and what happenedLast valid backup9:45Failure10:00Detection10:12Decision10:20Restored10:52Service normal11:05Actual RPO 15 minActual RTO 65 minData lossAgreed: 15 min · Actual: 15 minMeets the objective.DowntimeAgreed: 60 min · Actual: 65 minDoes not meet it: the design is revised.It was detected after 12 minutes: half the overrun was in detection, not in the restore.Illustrative data; they do not correspond to any real caseN100F · n100f.com

  • Recovery point objective (RPO): how much data can be lost, measured in time. If it is fifteen minutes, you need to back up or replicate at least every fifteen minutes.
  • Recovery time objective (RTO): how long the service can be down. If it is one hour, the full recovery (detect, decide, restore, verify) has to fit inside an hour.

Each value has a cost: the smaller they are, the more redundancy and the more monthly spend. That is why they are agreed with the business and written down, like the rest of the quality requirements.

From the number to the design

Objective Usual design
Hours of downtime and a day of data acceptable Daily backups and documented manual restore
One hour of downtime, minutes of data Frequent backups, replicas and automated recovery in the same region
Minutes of downtime, no practical loss Synchronous replicas and deployment across several zones or regions, with automatic failover

What is not tested does not exist

A backup that has never been restored is an assumption. A recovery plan that has never been executed is a document. Recovery is tested by:

  1. Actually restoring, periodically, in a separate environment, and measuring how long it takes.
  2. Simulating controlled failures (an instance, a zone, a dependency) to check that the system reacts as designed.
  3. Rehearsing the procedure with the people who will carry it out, not only with whoever wrote it.

Every test leaves a written result: actual recovery time, data lost and what had to be corrected. If the actual time exceeds the objective, the design does not meet its requirement, even if it has never failed.

How we apply it: Cloud Architecture.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk