N100FN100F
Published on

Procedures that run themselves

A procedure written in a document depends on somebody finding it, reading it properly and not skipping a step at three in the morning. The same procedure as code runs the same way every time, and calls a person only when judgement is needed.

Service line Automation and IaC

The document nobody finds in time

Every operation has procedures: what to do when a disk fills up, how to renew a certificate, how to restart a service that has hung. They usually live in documents. At best, up to date. Usually, written years ago by somebody who is no longer around.

During a night-time alert, whoever is on call has to find the document, understand it and run it by hand without making a mistake. It is exactly the moment when all of that is done worst.

From document to code

A procedure that runs itselfAutomatic alert, diagnosis, fix and verification, escalating to a person if it is not resolved.IAC · OPERATIONSA procedure that runs itselfAlertdisk at 90%Diagnoseis it the known case?Fixclears and expands ≤ limitVerifydisk < 70%?ResolvedrecordedyesEscalate to a personwith the diagnosis donenounexpected caseGOOD CANDIDATESClear temporary filesRenew certificatesRotate logsRestart known faultIf it triggers every week, you do not automate the patch better:you resolve the cause.N100F · n100f.com

A procedure as code does the same as the document, but without depending on who runs it:

  1. It triggers itself, from the alert or from a schedule.
  2. It diagnoses before acting: it checks that the situation is the one the procedure knows how to resolve.
  3. It applies the known fix, repeatably (if it is already applied, it does nothing).
  4. It verifies the result: was it actually resolved?
  5. It escalates to a person if the verification fails or if the case is not the expected one, handing over what it has already diagnosed.

And every run is recorded: when it triggered, what it found, what it did and with what result.

Where to start

The best candidates are tasks that are frequent, well understood and low risk: clearing temporary files, renewing certificates, rotating logs, restarting a service with a known fault, expanding a disk within a limit. Each saves little on its own; together they free up many hours and, above all, many nights.

What does not change

An automated procedure does not replace analysis. If the same problem triggers every week, the answer is not to automate the patch better, but to resolve the cause (the idea behind every failure leaves a rule).

And not every procedure should be automated. Knowing what to leave out matters as much as everything above.

How we apply it: Automation and IaC.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk