Data lineage: what it is, why it matters and a worked example

What is data lineage?

Data lineage is the record of where a piece of data came from, what happened to it along the way, and where it ended up. For a number on a dashboard, lineage answers: which events or source tables feed it, which jobs transformed them, which filters and definitions were applied, and which other reports reuse it.

Lineage can be captured at different levels of detail:

• Table or dataset level: this dashboard reads table B, which is built from tables A1 and A2.
• Column or field level: the revenue column comes from order_total minus refund_amount, converted to USD.
• Business level: the "Weekly active users" metric means people with at least one qualifying event in the last 7 days, owned by the product analytics team.

Why data lineage matters

A worked example: tracing a metric

Lineage for "Weekly active users (web)" in a typical analytics stack (sample):

When WAU drops, walk the table from the bottom up: was the definition or filter changed? Did the daily roll-up exclude something? Did fewer raw events arrive, and if so, were they rejected at the collector or never sent? Each step either explains the drop or rules itself out.

Data lineage vs related ideas

• Data provenance is narrower: the origin and history of a specific record, often for audit or science. Lineage covers the whole flow.
• Data dictionary defines what each field means; lineage shows how fields connect.
• Data observability monitors freshness, volume and schema in production; lineage tells you where an anomaly came from and what it affects.
• Data governance sets rules and ownership; lineage is one of the tools that makes those rules checkable.

How lineage is captured

1. Parsing code and queries. SQL and transformation code reveal which tables and columns feed which.
2. Reading platform metadata. Warehouses, orchestration tools and BI tools log reads and writes.
3. Declaring it in models. Transformation frameworks build a dependency graph from the models you write.
4. Documenting it manually. For tracking, the tracking plan is the upstream end of lineage: which event comes from which screen, platform and release.

Automated lineage is accurate for code it can parse; it misses manual steps such as spreadsheets and copy-paste. Plan for both.

Common mistakes

• Treating lineage as a diagram project. A graph nobody uses during incidents goes stale in weeks. Start from the three or four metrics leadership looks at.
• Stopping at the warehouse. Many data problems start at collection: tracking code, consent, blocking. Include the tracking layer.
• No owners. Every node in the lineage needs a person or team to ask.
• Ignoring definitions. Two dashboards can share every table and still disagree because one counts sessions and the other counts people.

In TapCub

TapCub does not draw a lineage graph, but it exposes the steps where analytics data is most often lost. The tracking plan records each event's platforms, trigger, properties, owner and status from proposed to live. The data quality page shows ingestion warnings, collector rejects by reason (domain mismatch, exclusion rule, quota, rate limit, invalid payload, bots), blocked properties and events that stopped arriving. When a metric moves unexpectedly, those two pages cover the top of the lineage table above. For the rest of a pipeline, see data pipeline and data quality.

Sources

• IBM. What is data lineage?
• OpenLineage. An open standard for lineage metadata collection.
• DAMA International. DAMA-DMBOK: Data Management Body of Knowledge, 2nd ed. (metadata management chapter).