Drift always arrives
However much discipline there is, reality moves away from the code: an emergency change made by hand, an automatic adjustment by the provider, an update that modified a setting. That distance is called drift.
Its existing is not a failure. Not knowing about it is. Undetected drift is what causes an innocent code change to 'undo' an emergency fix nobody carried over, or a rebuilt-from-code environment not to behave like the original.
Checking on a schedule
Detection is done periodically and automatically:
- For the infrastructure: the plan is calculated against the environment without applying it. If the plan is not empty, there is drift.
- For the servers: each one is checked against its baseline, as we saw in the previous article.
The more frequent the check, the smaller the differences and the easier it is to know where they came from.
Every difference, a decision
Detecting is not enough: every difference requires choosing between two paths.
- Reality was right. The change by hand was necessary (an emergency adjustment, for instance). It is taken into the code, reviewed and applied formally.
- The code is right. The change by hand should not have existed. The code is applied again to return reality to what is declared.
What will not do is the third option, the most common one: leaving it alone. Every ignored drift makes the next one harder to understand.
Reducing the source
The best way to have little drift is to leave little room for creating it: write permissions in the console for emergencies only, with a record of who used them, and the rule that every emergency change is carried into the code that same day.
With reality and the code aligned, you can face what frightens people most: changing important things in production.
How we apply it: Automation and IaC.