N100FN100F
Published on

Real rollback

Every change plan says how to move forward; few say how to go back. In infrastructure, rolling back is not always undoing: sometimes it is keeping the old version alive until you are sure of the new one.

Service line Automation and IaC

'If it goes wrong, we will undo it'

It is the sentence that reassures people before an important change, and the one least often honoured. Undoing an application code change is usually easy. Undoing an infrastructure change is not always: a migrated database, a reconfigured network or a replaced certificate do not come back with one click.

That is why, in infrastructure, rollback is designed before the change, not improvised afterwards.

Keeping the old version alive

The old one stays alive until we are sureCurrent version and new version in parallel, with the five steps for switching traffic and being able to go back.IAC · ROLLBACKThe old one stays alive until we are sure TrafficCurrent versionkeeps serving New versionbrought up alongside and tested 1Bring up alongside2Test with no traffic3Switch the traffic4If it fails, back in seconds5Retire the old one once confirmedNo important change without a written answer to:'how do we roll back, and how long does it take?'.N100F · n100f.com

The safest technique is not to modify what works, but to bring the new version up alongside it:

  1. The current version keeps serving.
  2. The new version is created, complete, in parallel.
  3. It is tested with no real traffic or with a small share.
  4. Traffic is switched to the new one.
  5. If something fails, traffic returns to the previous one in seconds, because it is still there.
  6. Only once the new one is confirmed is the previous one retired.

It costs more for a few hours (there are two versions at once) and saves a great deal when something goes wrong.

When you cannot have two versions

Some changes do not allow parallel running, especially with data. For those:

  • A verified backup immediately beforehand, restorable and tested (not an assumption).
  • Changes in two stages: first the new is added without removing the old (a column, a field) and only afterwards, once everything uses the new, is the old removed.
  • A written window and plan: what gets done, how long it takes, at what point there is no going back and who decides to continue.

The rule

No important change is approved without a written answer to 'how do we roll back, and how long does it take?'. If the answer is 'we cannot', the change needs a different shape.

Rolling back protects you from errors that have already happened. Better still is preventing certain errors from ever being applied.

How we apply it: Automation and IaC.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk