N100FN100F
Published on

Architecture of a cost pipeline

From the provider's raw file to data you can decide on there are six layers, and each one exists for a reason. This is how you design a cost pipeline that can be audited and that depends on nobody.

Service line FinOps Engineering

Why layers and not a single step

The temptation is to write one transformation that reads the provider's export and delivers the final report. It works until the first error: when a number does not add up, there is no way to tell where it went wrong.

A layered pipeline solves that. Each layer has a single responsibility, stores its result and can be reviewed on its own. If something fails, you know which layer, and you reprocess from there without touching anything upstream.

The six layers

What happens to the data in each layerSix pipeline layers with a sample of how the data changes in each one and its illustrative row count.FINOPS · COST PIPELINEWhat happens to the data in each layer01RawProvider's file, untouched1,204,000 rows02NormalisedFOCUS columns, same meaning1,204,000 rows03Corrected−23,500 duplicates · commitments spread1,180,500 rows04AtomicResource × day × charge type3,412,000 rows×305AllocatedEvery row with team and application94% allocated06ConsumptionTables ready for decisions12 tablesIllustrative data; they do not correspond to any real caseN100F · n100f.com

1. Raw. The provider's export exactly as it arrives, untouched. It is always kept, because it is the evidence: if an error of interpretation comes to light tomorrow, you reprocess from here.

2. Normalised. The data translated into FOCUS. Each provider has its own translation, but all of them produce the same format. From this layer on, the pipeline neither knows nor cares which provider each row came from.

3. Corrected. This is where you fix what the source data brings through wrong or ambiguous: duplicate rows, commitments that do not properly reflect the usage they applied to, misclassified charges. Each correction is a written rule with its reason, not a manual tweak.

4. Atomic. The data taken down to its smallest useful unit: one row per resource, per period and per charge type, with list, billed and effective cost. It is the layer everything else derives from, and the one used to reconcile with the bill.

5. Allocated. Each cost attributed to an owner: team, application, product. It includes the rules for spreading shared costs and whatever carries no tag.

6. Consumption. Wide tables, prepared to answer specific questions: cost per product and month, cost per unit of business, variance against budget. This is what the dashboards and reports read.

Design principles

  • Raw data is never modified. Everything else can be regenerated from it.
  • One responsibility per layer. If a layer normalises and allocates, when it fails you will not know which of the two things failed.
  • Written rules, not manual tweaks. Every correction and every allocation rule lives in the code or the configuration, with the reason beside it.
  • Reconciliation on every lap. The period total in the atomic layer has to match the bill. If it does not, the pipeline stops before publishing.
  • Reprocessable. Any period can be recalculated end to end and must give the same result.

With that discipline, the pipeline stops being one person's black box and becomes a system anyone on the team can audit six months later.

How we apply it: FinOps Engineering.

If your cloud cost keeps growing and nobody knows why, or your architecture depends on whoever built it, let’s talk.

Let’s talk