Published on

Before fixing, know what is there

Every inherited system has a documented part and another that exists only in production. How to run a survey that uncovers what is really running before changing anything.

Service line Cloud Architecture

What the paper says and what production says

Whoever inherits a system usually receives some documentation: a diagram, some notes, perhaps a design document from years ago. What they almost never receive is the part that matters: the scheduled jobs nobody remembers, the dependencies between systems that appear in no drawing, the access that was granted 'temporarily' and is still there.

Changing something without knowing that part is the fastest way to cause an incident.

The submerged part

What is documented and what is runningAn illustrative iceberg: what is documented above, what exists only in production below.ARCHITECTURE · SURVEYWhat the paper says and what is running▲ documented▼ only in productionDiagramMain servicesDatabaseNightly scheduled jobs14Implicit dependencies23Credentials with no owner9Orphaned resources31Configuration made by hand17Documented≈ 30 %Found by the survey: ≈ 70%Illustrative data; they do not correspond to any real caseN100F · n100f.com

What usually surfaces when a survey is done:

  • Scheduled jobs that move data or clean up tables every night.
  • Implicit dependencies: a service that calls another at an address written into a configuration file.
  • Access and credentials with no owner and no expiry.
  • Orphaned resources that are still costing money.
  • Configurations made by hand that are in no code anywhere.

How it is done

  1. Inventory from the real source, not from the documentation: the provider's APIs, the network configuration, the access logs and the billing all say what exists.
  2. Dependency map: who talks to whom, by observing the traffic and the configuration, not by asking.
  3. Comparison against what is documented: every difference is noted. It is the most valuable list the exercise produces.
  4. A list of blockers, with their size: what prevents each part from being changed or migrated, and how much effort it takes to resolve.
  5. A living document: the result is left as documentation as code, so the survey does not have to be repeated.

What not to do during the survey

Fix things. It is tempting, but every change during the survey moves the ground you are measuring. First you understand everything; then you decide what to touch, in what order and at what risk.


How we apply it: Cloud Architecture.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk