About OpenLineage

OpenLineage is an open framework for data lineage collection and analysis. At its core is an extensible specification that systems can use to interoperate with lineage metadata.

Design

OpenLineage is an Open Standard for lineage metadata collection designed to record metadata for a job in execution.

The standard defines a generic model of dataset, job, and run entities uniquely identified using consistent naming strategies. The core model is highly extensible via facets. A facet is user-defined metadata and enables entity enrichment. We encourage you to familiarize yourself with the core model below:

How OpenLineage Benefits the Ecosystem

Below, we illustrate the challenges of collecting lineage metadata from multiple sources, schedulers and/or data processing frameworks. We then outline the design benefits of defining an Open Standard for lineage metadata collection.

BEFORE:

• Each project has to instrument its own custom metadata collection integration, therefore duplicating efforts.
• Integrations are external and can break with new versions of the underlying scheduler and/or data processing framework, requiring projects to ensure backwards compatibility.

WITH OPENLINEAGE:

• Integration efforts are shared across projects.
• Integrations can be pushed to the underlying scheduler and/or data processing framework; no longer does one need to play catch up and ensure compatibility!

Scope

OpenLineage defines the metadata for running jobs and their corresponding events. A configurable backend allows the user to choose what protocol to send the events to.

Core model

A facet is an atomic piece of metadata attached to one of the core entities. See the spec for more details.

Spec

The specification is defined using OpenAPI and allows extension through custom facets.

Integrations

OpenLineage supports integrations with several systems.

• Apache Airflow
• Apache Flink
• Apache Hive
• Apache Spark
• dbt
• Feast
• Great Expectations
• SQL
• Trino

Related projects

• Marquez: Marquez is an LF AI & DATA project to collect, aggregate, and visualize a data ecosystem's metadata. It is the reference implementation of the OpenLineage API.
  • OpenLineage collection implementation
• Egeria: Egeria Open Metadata and Governance. A metadata bus.

Contributing to OpenLineage

OpenLineage is an LF AI & Data Foundation Graduate project under active development, and we welcome contributions.

See CONTRIBUTING.md for more details about how to contribute.