N100FN100F
Published on

Every failure leaves a rule

An incident resolved that changes nothing will happen again. The way to learn from failures is to turn each one into something permanent: a test, a validation, a rule or a written decision.

Service line Platform Engineering

Resolving is not learning

A service fails, the team fixes it, everybody breathes and goes back to what they were doing. Three months later, the same failure, in another service, with another person on call who did not know what happened the first time.

Resolving an incident is necessary. But if at the end of the incident the platform is exactly as it was before, the only thing gained was time.

The full cycle

Every failure leaves a ruleThe incident cycle through to a permanent action that goes back into the platform.PLATFORM · LEARNINGEvery failure leaves a ruleFailureRecoverBlameless analysisRoot causePermanent actiongoes back intothe platform'PERMANENT ACTION' MEANS ONE OF THESE A test A validation A rule A written decision'Being more careful' does not count: it stops working once everybody forgets.N100F · n100f.com

  1. Recover the service, with the procedure designed for it (we covered this in designing for failure).
  2. Analyse without looking for culprits. The question is not who got it wrong, but what allowed the error to reach production.
  3. Find the root cause, which is almost never the obvious one. 'Somebody pushed a bad configuration' is the symptom; 'nothing was checking that configuration before deploying it' is the cause.
  4. Turn it into something permanent.

What counts as 'permanent'

'Being more careful' does not count. What counts is something that keeps working once everybody has forgotten the incident:

  • A test that would have caught the failure.
  • An automatic validation in the build, like the ones in the article on letting the build fail.
  • A rule in the organisation's standard, with its check.
  • A decision record if the failure revealed that a design decision no longer holds.

Measuring it

A platform that learns shows it in two numbers: repeat incidents go down and more and more failures stop at the build instead of reaching production.

Many of those rules and validations are applied today by AI assistants working inside the platform. How that is governed is the last block of the series.

How we apply it: Platform Engineering.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk