N100FN100F
Published on

What is best not automated

Automating does not always pay off. Some tasks are so rare, so variable or so risky that the script costs more than it saves, or does more damage than it prevents. Telling them apart is part of the craft.

Service line Automation and IaC

The temptation to automate everything

Once you have seen what automation saves, the opposite temptation appears: automating anything done more than once. And some automations end up worse than the manual work: they take longer to build than the time they will save in years, they fail when the situation changes slightly, or they execute with complete confidence something that needed somebody to think about it.

Two questions for deciding

What to automate, and what not toA frequency-against-risk matrix with four answers: automate, automate with approval, document and keep manual.IAC · JUDGEMENTWhat to automate, and what not toAutomate with approvalfrequent and riskyKeep manual and rehearsedrare and riskyAutomate completelyfrequent and low riskWritten procedurerare and low riskFrequentRareRiskyor variableLowriskExamples: clearing temporary files (green) · destructive network change (amber) ·annual migration (blue) · recovery from a total disaster (red).N100F · n100f.com

Before automating a task it is worth answering two questions: how often does it happen? and how much risk or variation does it carry each time?

  • Frequent and low risk: automate completely. This is the territory of the previous article.
  • Frequent but risky: automate the preparation and the execution, but with human approval before the irreversible step (like the plan with approval for destructive changes).
  • Rare and low risk: a good written procedure is probably enough. Automating something that happens once a year rarely pays off.
  • Rare and risky: keep it manual, written in detail and rehearsed. Recovery from a total disaster usually sits here: its pieces are automated, but the decision and the coordination belong to people.

Signs that something should not be automated yet

  • Nobody can describe precisely when the task is 'done properly'.
  • Every time it is done, it is done differently.
  • An error cannot be undone.
  • It requires deciding something that is not written in any rule.

In all these cases, the first step is not a script, but writing the procedure down and repeating it until it stops varying. What can be written down precisely can be automated afterwards.

Closing the series

Rebuilding from code, declaring instead of ordering, looking after the state, reusing with modules, reading the plan and its cost, keeping every server on its baseline, detecting drift, being able to roll back, turning standards into rules that check themselves, bringing in what you inherited and automating operations with judgement. Infrastructure stops being a collection of unique pieces and becomes something you can describe, review, repeat and explain.

How we apply it: Automation and IaC.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk