Data Lineage 2026 — Tools, Patterns, and Why It Matters

Sooner or later, a finance director asks where a number on a dashboard came from, and nobody in the room can answer without a week of digging. Data lineage is the record that turns that week into a few clicks: a map of which sources fed which tables, which transformations changed them, and which reports now depend on them. This article explains what lineage tracks, how tools capture it, what the major products actually record (including the limits their own documentation admits), and how to start without building a bureaucracy.

What data lineage tracks

Think of a restaurant's supply records. A diner finds something wrong in a dish, and the kitchen needs to answer three questions. Which supplier delivered the ingredient (backward)? Which other dishes used the same batch (forward)? And what did the cooks do to it between the delivery door and the plate (transformation)? Data lineage answers the same three questions about information instead of food.

The general definition is simple: lineage tracks how data is generated, transformed, transmitted, and used across systems over time, and it underpins impact analysis and the debugging of data pipelines (Data lineage, Wikipedia). In practice, teams use it for four jobs:

Question	Direction	Typical use
Where did this value come from?	Upstream	Debugging a wrong number, answering an auditor
What breaks if I change this?	Downstream	Impact analysis before renaming or dropping a column
What happened to the data on the way?	Transformation	Reviewing the SQL or code between source and report
Who or what consumed this table?	Downstream	Retiring unused assets, notifying owners

Four levels of detail

Lineage can be recorded at different resolutions, and the resolution decides how useful it is:

• Dataset level. Table A feeds table B. This is the easiest to capture and the least informative when something goes wrong, because a table can have 80 columns.
• Column level. Column revenue in table B is computed from quantity and unit_price in table A. This is where debugging gets fast.
• Job level. Which scheduled run, in which orchestrator, produced the output, and did it succeed.
• Code level. The exact SQL or script that did the work.

Column-level lineage is the capability that separates a pretty diagram from a working tool. A table-level graph tells you that fifteen upstream tables might be involved in a wrong figure. A column-level graph tells you which two.

A worked example

The following SQL builds a daily revenue table. Everything a column-level lineage tool records about it can be read straight off the statement:

create table analytics.daily_revenue as
select
    o.order_date,
    sum(oi.quantity * oi.unit_price) as revenue
from analytics.orders o
join analytics.order_items oi on oi.order_id = o.order_id
where o.status = 'completed'
group by o.order_date;

Output column	Source columns	Relationship
order_date	orders.order_date	Direct, passed through unchanged
revenue	order_items.quantity, order_items.unit_price	Direct, aggregated with sum
(both)	orders.order_id, order_items.order_id	Indirect, through the join
(both)	orders.status	Indirect, through the where filter
(both)	orders.order_date	Indirect, through group by

The last three rows matter more than they look. If someone redefines what status = 'completed' means, the revenue column changes even though no column of revenue was edited. A tool that tracks only the direct rows misses that dependency entirely, and this distinction between direct and indirect influence turns out to be built into the OpenLineage standard, covered below.

How lineage gets captured

There are four basic ways to learn how data moved, and every product mixes them.

1. Parse the SQL. A tool reads the query text and works out which columns feed which. It needs no changes to your pipelines, and it works on history. Its weakness is that it only understands what it can parse. Dynamic SQL assembled at runtime, vendor-specific syntax, and logic hidden in Python are all harder.

2. Instrument the jobs. The orchestrator or processing engine reports, while it runs, what it read and wrote. This captures what actually happened, including code paths that SQL parsing cannot see, but it requires an integration for every engine.

3. Read system metadata. Warehouses keep query logs and dependency tables. Collecting them gives broad coverage cheaply, within whatever the vendor decides to record.

4. Annotate by hand. People draw the edges that tools miss, such as a spreadsheet someone emails every Monday.

An analogy helps here. Parsing is reading a recipe and predicting what the kitchen will do. Instrumentation is a camera over the stove that records what the cook did. Metadata collection is the receipt printer. Manual annotation is the cook's handwritten note on the back. Each answers the question differently, and mature setups combine them.

The vendors' own descriptions confirm the mix. Atlan says it builds lineage from SQL parsing of queries in Snowflake, BigQuery, Redshift, and Databricks, native integrations that read tool APIs, OpenLineage events from Airflow, Spark, and dbt Cloud, and custom lineage through APIs or a visual builder (Atlan). Alation lists metadata harvesting, SQL log analysis, query parsing, and manual annotation (Alation). DataHub extracts lineage from ingestion sources that support it, lets users add or remove edges in the interface since version 0.9.5, and accepts edges emitted through its API (DataHub docs).

OpenLineage: a standard for the events

OpenLineage is an open framework for collecting lineage, built around a specification that lets different systems share the same kind of metadata. It is a Graduate project of the LF AI & Data Foundation, and its documentation lists integrations for Apache Airflow, Apache Flink, Apache Hive, Apache Spark, dbt, Feast, Great Expectations, SQL, and Trino (OpenLineage docs).

The model has three core entities (OpenLineage object model):

• A job is a process that consumes or produces datasets: a workflow task, a query, a model, or a validation checkpoint.
• A run is one execution of a job, identified by a client-generated UUID. A daily script produces a new run each day.
• A dataset is a discrete collection of data, such as a table or a storage object, identified by a namespace and a name derived from where it physically lives.

Events carry run state changes (usually starting with START and ending with COMPLETE), and each can list the inputs and outputs of the run. Extra detail travels in facets, which are named blocks of metadata attached to jobs, runs, or datasets. Facets are where OpenLineage gets rich. Datasets can carry a schema, input datasets can carry dataQualityMetrics and dataQualityAssertions, and output datasets can carry a column lineage facet.

The column lineage facet

The column lineage facet maps each output column to its input columns. Each link carries a transformation with a type of DIRECT (the output value is derived from the input) or INDIRECT (the input only influences the output, as with a filter or sort). Subtypes get specific: direct ones include IDENTITY, TRANSFORMATION, and AGGREGATION, and indirect ones include JOIN, GROUP_BY, FILTER, SORT, WINDOW, and CONDITIONAL. A masking flag records whether an input was obfuscated, for instance by hashing (column lineage facet spec).

Here is an abridged event for the daily revenue job above. Fields such as the producer and schema URL are omitted for readability:

{
  "eventType": "COMPLETE",
  "eventTime": "2026-10-21T02:15:00Z",
  "run": { "runId": "01933e0c-6b6a-7d10-9a3e-5b1f0c2d4e11" },
  "job": { "namespace": "analytics-prod", "name": "build_daily_revenue" },
  "inputs": [
    { "namespace": "warehouse", "name": "analytics.orders" },
    { "namespace": "warehouse", "name": "analytics.order_items" }
  ],
  "outputs": [{
    "namespace": "warehouse",
    "name": "analytics.daily_revenue",
    "facets": {
      "columnLineage": {
        "fields": {
          "revenue": {
            "inputFields": [
              { "namespace": "warehouse", "name": "analytics.order_items", "field": "quantity",
                "transformations": [{ "type": "DIRECT", "subtype": "AGGREGATION" }] },
              { "namespace": "warehouse", "name": "analytics.order_items", "field": "unit_price",
                "transformations": [{ "type": "DIRECT", "subtype": "AGGREGATION" }] },
              { "namespace": "warehouse", "name": "analytics.orders", "field": "status",
                "transformations": [{ "type": "INDIRECT", "subtype": "FILTER" }] }
            ]
          }
        }
      }
    }
  }]
}

A consumer of these events can be anything that speaks the format. Marquez, which WeWork released as open source, is the reference implementation: a metadata server with an OpenLineage-compatible endpoint, a web interface for the dependency graph, and a lineage API for queries such as root cause analysis (Marquez). Google's catalog can also import OpenLineage events, and Atlan lists them as an input. The practical value of a standard is that the choice of where to store lineage can be made later, without rewriting how it is collected.

Why lineage matters more in 2026

Lineage has been useful for as long as pipelines have broken. What changed is how many people besides engineers ask for it.

Regulated reporting. After the 2007-2009 financial crisis, the Basel Committee on Banking Supervision published its principles for risk data aggregation and risk reporting (BCBS 239) in January 2013, aimed in particular at the largest global banks (Bank for International Settlements). Banks have since used lineage as one way to show where reported risk figures came from; Wikipedia's lineage article cites BCBS 239 as an example of a regulation lineage can support.

AI training data. Article 10 of the EU AI Act sets data governance requirements for high-risk AI systems, including documenting data collection and origin, data preparation steps such as annotation, cleaning, and aggregation, and examining datasets for bias. It says datasets should be relevant, sufficiently representative, and, to the best extent possible, free of errors and complete. According to the consolidated text on artificialintelligenceact.eu, the rules apply from December 2, 2027 for Annex III systems and August 2, 2028 for Annex I systems (AI Act Article 10). Lineage does not by itself make a system compliant, but it is the natural way to answer "where did this training set come from and what was done to it?" Our explainer on AI data provenance covers that question for training and retrieval data.

Trust in dashboards. Executives and analysts ask how a figure was calculated. If the answer takes days, they stop trusting figures.

Incident response. When a source schema changes overnight, forward lineage lists every downstream asset that may be affected. Without it, you find out from the people whose dashboards broke.

Cost control. Lineage reveals duplicated pipelines and tables nobody reads. Alation's page makes this point about duplicate reports and processes, and Monte Carlo's lineage page gives an example of a data scientist who spent time merging and cleaning data without noticing that a downstream table already combined the same sources.

A caution against overselling: published figures such as "teams report 30 to 50 percent faster debugging" circulate in vendor material, but none of the sources checked for this article supports a specific number, so this guide does not repeat one. The benefit is real and local. Measure how long your last three data incidents took to diagnose, then see whether lineage would have shortened each.

What the major tools record, and where they stop

Every lineage product has a scope, a retention window, and a list of things it cannot see. The details below come from each vendor's own documentation, read in October 2026.

dbt

dbt builds a dependency graph from ref() and source() calls in your project. The dbt Catalog renders it as an interactive DAG with filters and lenses that overlay materialization type, last run status, or test status (dbt Catalog). Column-level lineage is a separate capability with conditions attached. In the dbt platform it requires an Enterprise or Enterprise+ plan. Locally, it requires dbt v2 with strict static analysis, available through the VS Code extension, dbt Docs v2, or the command line (dbt column-level lineage).

The documented limitations are instructive:

• Only select logic is tracked. Columns used in joins or filters do not appear as dependencies, which is exactly the indirect influence from the worked example above.
• Lineage relies on SQL parsing, which can fail on ambiguous or complex SQL, such as complex lateral joins or JSON unpacking.
• Python models break the chain, because their lineage cannot be parsed.
• Hardcoded table names instead of ref() can prevent lineage from being determined.

That last point is the kind of discipline problem no tool fixes for you. For the foundation of how dbt projects are structured, see our dbt tutorial for analysts.

Databricks Unity Catalog

Unity Catalog captures table-level and column-level lineage automatically for queries run on Databricks. Notebooks, jobs, and dashboards that read or write tables appear in the graph, along with pipelines and some model-serving assets. Its documentation lists real constraints: only data captured after September 1, 2024 is available, renamed catalogs, schemas, tables, or columns do not preserve lineage, and user-defined functions obscure column mappings. The lineage system tables keep a rolling one-year window (Unity Catalog lineage). Because Delta Lake, MLflow, Apache Spark, and Unity Catalog originated with Databricks, this tool is the natural fit for teams already centered on that platform. Our Apache Spark explainer describes the engine underneath.

Snowflake

Snowflake's lineage in Snowsight requires Enterprise Edition or higher and shows upstream and downstream objects one step at a time, including column-level lineage. Object and column lineage are each retained for one year, and lineage launched in November 2024, so data movement before that date is not available. Its documented exclusions include shared databases, the SNOWFLAKE database and INFORMATION_SCHEMA objects, tables used only for filtering or joining, and data moved by separate, disjointed queries within stored procedures (Snowflake lineage).

Google Cloud: Knowledge Catalog, formerly Dataplex Universal Catalog

Google's lineage models processes, runs, and events, and for BigQuery it covers tables and views touched by copy jobs, load jobs, and query jobs including DDL and DML. Column-level lineage is supported for top-level columns with caveats: it is skipped when a job creates more than 1,500 column-level links, is not kept through ALTER TABLE RENAME, and does not support nested fields such as STRUCT or JSON. Lineage is kept for only 30 days (Google Cloud data lineage).

The retention gap: Google keeps lineage for 30 days. Snowflake and Databricks system tables keep it for a year. An audit question that arrives six months after a change can be answered by two of those three, unless you export the records yourself.

Catalogs and observability platforms

DataHub began at LinkedIn in 2019, is licensed under Apache 2.0, and has a managed offering from the company founded by its creators (DataHub on GitHub). Its lineage covers datasets, columns, and data pipeline tasks, and it offers impact analysis. Its documentation also warns that manual and automated lineage can overwrite each other.

Atlan says it provides column-level lineage across more than 80 systems and describes itself as scoring highest for lineage and impact analysis in the 2025 Gartner Magic Quadrant for Metadata Management Solutions. That is the vendor's description of a third-party report, and its page states no pricing.

Alation emphasizes business-facing lineage maps with overlays for data quality and trust flags, plus auditable records of how data is transformed and used.

Monte Carlo, a data observability platform, includes lineage as a view of table dependencies and uses it to connect related incidents to the objects they affect. For more on the monitoring side, see our guide to data quality monitoring.

Tool	Granularity documented	Capture method	Retention or scope note
dbt Catalog	Model level; column level on Enterprise plans	Static analysis of the project	Select logic only; Python models break the chain
Unity Catalog	Table and column	Automatic from Databricks queries	Data since Sept. 1, 2024; system tables keep one year
Snowflake	Object and column	Automatic from Snowflake activity	Enterprise Edition or higher; one year
Google Knowledge Catalog	Table, view, top-level column	Automatic from BigQuery jobs, plus OpenLineage import	30 days
DataHub	Dataset, column, pipeline task	Ingestion, API, manual edits	Open source; self-hosted or managed
Atlan	Column, per vendor	SQL parsing, integrations, OpenLineage, custom	Pricing not published on its lineage page
Alation	Technical plus business lineage	Harvesting, query logs, parsing, manual	Vendor documentation
Marquez	Job and dataset, OpenLineage native	OpenLineage events	Reference implementation, open source

Putting it together: five implementation patterns

The right design depends on where your transformations run.

Pattern 1: the project graph only. If your transformations are almost all in dbt and the team is small, the dbt graph answers most questions. Document sources clearly and keep references going through ref() so the graph stays complete.

Pattern 2: orchestrator events. Have Airflow, Spark, or dbt emit OpenLineage events and store them in Marquez, DataHub, or a catalog. This captures runtime behavior, including steps that SQL parsing cannot see. For how orchestrators differ in what they emit and how they schedule work, see our Airflow, Prefect, and Dagster comparison.

Pattern 3: a commercial catalog. Connect your sources to a platform such as Atlan or Alation and treat lineage as one feature among governance, glossaries, and ownership. This suits organizations that need business users to browse lineage, not only engineers.

Pattern 4: an open-source catalog. DataHub offers a comparable scope for teams with the engineering capacity to run it, or to buy its managed version.

Pattern 5: warehouse-native lineage. Unity Catalog, Snowflake lineage, and Google's catalog require little setup if one platform holds most of your data. The trade-off is blind spots wherever data leaves that platform, and the retention windows above.

Most real setups end up as a blend: warehouse-native lineage for what happens inside the warehouse, dbt for the transformation layer, and OpenLineage events from orchestrators to stitch the rest together. A broader view of how these pieces sit in a stack is in our modern data stack reference.

Lineage and its neighbors

Lineage is often confused with adjacent disciplines. Separating them avoids buying one tool to solve another's problem.

Catalog. A catalog is a searchable directory of data assets: names, descriptions, owners. Lineage is the dependency graph between them. Modern catalogs include lineage, and lineage tools tend to include light cataloging, so the boundary blurs in products even where it stays sharp in concept.

Data contracts. A contract is an agreement between a producer and its consumers about a dataset's schema and behavior. Lineage shows who the consumers are. A contract without lineage is a promise to unknown people, and lineage without contracts shows you the damage after a change, not before it. Our guide to data contracts covers the producer-consumer side.

Quality monitoring and observability. Quality checks tell you that a table looks wrong. Lineage tells you where the wrongness probably began and what it touched. OpenLineage even has facets for quality metrics and assertions on datasets, a sign that the two are meant to meet. Our explainer on AI observability shows the same pairing for models and agents.

Machine learning lineage. Models add their own questions: which training set and code version produced this model, and which features does it read? Experiment trackers and registries handle much of this, and our MLOps tools comparison covers that category. A feature store is another place where lineage is recorded, because features are derived data with consumers. When a model starts degrading, forward and backward lineage helps separate a data change from model drift in production, and tracing training data back to its sources is how teams investigate data leakage that inflates evaluations.

What goes wrong in lineage projects

Lineage initiatives tend to fail in a handful of repeatable ways.

1. No question to answer. "We should have lineage" is not a requirement. Write down the three questions it must answer: an audit question, a debugging question, an impact question. Success is then measurable.
2. Stopping at table level. A table-level graph is a start, and its answers are too coarse for real debugging. Plan for columns.
3. Ignoring the indirect edges. If a tool tracks only direct column derivation, changes to filters and joins will surprise you. Check how your tool handles them, using the worked example above as a test case.
4. Trusting silent gaps. The limitations lists above exist because lineage is incomplete by default: Python steps, dynamic SQL, renames, and stored procedures all create holes. An edge missing from the graph does not mean no dependency exists. Test your tool on one pipeline you understand fully before relying on it for one you do not.
5. Letting it rot. Catalogs and lineage graphs go stale without an owner. Assign one person or team, and review gaps regularly.
6. Mixing manual and automatic edges carelessly. DataHub's documentation warns these can overwrite each other. Decide which source of truth wins before you let people edit.
7. Forgetting retention. If you need lineage for an audit window longer than your tool's retention, export or archive it on a schedule.

Choosing a starting point

Your situation	Sensible first step
Small team, nearly everything in dbt	Use the dbt graph; keep references in ref() and source()
Mostly one platform (Databricks, Snowflake, BigQuery)	Turn on the native lineage and read its limitations page
Many orchestrated jobs across tools	Emit OpenLineage events and pick a store
Business users must browse lineage	Evaluate a catalog with business-facing lineage views
Strong DevOps, limited budget	Trial DataHub self-hosted
Regulated reporting or AI training data	Define the audit questions first, then check retention and export
Nothing today	Pick one critical dashboard and trace it end to end by hand, then see what a tool would automate

The last row is not a joke. Tracing a single important dashboard by hand teaches you what your stack hides, and it gives you a concrete test for any vendor demo. If a tool cannot reproduce a graph you drew yourself, you have learned something before spending anything.

Frequently asked questions

Q: How is lineage different from a data catalog? A: A catalog is a directory of data assets with descriptions and owners. Lineage is the graph of dependencies among them. Most modern catalogs include lineage, and most lineage products include some cataloging, so the practical difference is where the product's center of gravity lies.

Q: Can I add lineage to a legacy stack? A: Partly. SQL parsing works on query history for many warehouses, but legacy ETL tools, stored procedures, and spreadsheets often need manual or custom edges. Expect the first pass to cover the modern core and leave visible gaps at the edges, and use those gaps as a to-do list.

Q: Does lineage cover Python and Spark transformations? A: Sometimes. OpenLineage has an integration for Spark, and Unity Catalog captures lineage for queries using Spark DataFrames, but dbt's column lineage cannot trace through Python models and Databricks does not capture RDDs. Coverage for code is real but less complete than for SQL.

Q: Is column-level lineage worth the extra effort? A: For debugging and impact analysis, usually yes, because it narrows the suspects from whole tables to specific columns. Check how your tool treats filters, joins, and aggregations, since direct-only lineage misses a class of changes.

Q: How long is lineage kept? A: It depends on the product. Per vendor documentation read for this article, Google's catalog keeps lineage for 30 days, Snowflake keeps object and column lineage for one year, and Databricks lineage system tables keep a rolling one-year window. Export it if you need longer.

Q: How does lineage relate to data contracts and data quality? A: They complement each other. Contracts state what a dataset promises, quality checks test whether it delivered, and lineage shows who depends on it and where a problem started. Mature teams use all three, and linking them (for example, attaching quality assertions to lineage events) is where the value compounds.

Q: What should I do first if I have nothing? A: If you use dbt, publish its project documentation and read the lineage graph critically. If not, pick one critical report, trace it by hand, then try an open standard such as OpenLineage events from your orchestrator. Resist the urge to deploy an enterprise platform before you know which questions you need answered.

The takeaway

Lineage looks like a diagramming problem and behaves like a coverage problem. Every tool draws attractive graphs, and every tool has places where the graph is silently incomplete: Python steps, renames, filters, stored procedures, and anything older than its retention window. The teams that get value from lineage are the ones that start with specific questions, test the tool against a pipeline they already understand, and treat the documented limits as part of the design. Capture what you can automatically, standardize the events so the storage choice stays open, and keep a human on the hook for the gaps.

---

Read next:

• SQL for Data Science: Essential Queries to Master
• What Data Cleaning Actually Involves
• Best Data Science Tools in 2026