Skip to main content

Getting Started

info

This page is in need of some refactoring. Suggestions are welcome using the Edit link at the bottom.

This guide covers how you can quickly get started collecting dataset, job, and run metadata using OpenLineage. We'll show how to collect run-level metadata as OpenLineage events using Marquez as the HTTP backend, then explore lineage metadata via the Marquez UI.

Prerequisites

Before you begin, make sure you have installed:

info

In this guide, we'll be using Marquez as the OpenLineage HTTP backend and running the HTTP server via Docker.

Run Marquez with Docker

The easiest way to get up and running with Marquez is Docker. Check out the Marquez source code and run the ./docker/up.sh script:

$ git clone git@github.com:MarquezProject/marquez.git && cd marquez

$ ./docker/up.sh
info

Pass the --build flag to the script to build images from source, or --tag X.Y.Z to use a tagged image.

To view the Marquez UI and verify it's running, open http://localhost:3000. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run-level metadata of current and previous job runs, and much more.

Collect Run-Level Metadata

Marquez is an LF AI & DATA incubation project to collect, aggregate, and visualize a data ecosystem’s metadata. Marquez is the reference implementation of the OpenLineage standard.

In this example, we show how you can collect dataset and job metadata using Marquez, Using the LineageAPI.

When you submit a lineage event, you first need to define an unique run ID that would look something like d46e465b-d358-4d32-83d4-df660ff614dd. This is usually in UUID format, and should be unique. This run ID will enable the tracking of run-level metadata over time for a job which may have a name, like my-job. So, let's get started!

info

The example shows how to collect metadata via direct HTTP API calls using curl. But, you can also get started using our client library for Java or Python.

Step 1: Start a Run

Use the run ID d46e465b-d358-4d32-83d4-df660ff614dd to start the run for my-job with my-input as the input dataset:

REQUEST

$ curl -X POST http://localhost:5000/api/v1/lineage \
-H 'Content-Type: application/json' \
-d '{
"eventType": "START",
"eventTime": "2020-12-28T19:52:00.001+10:00",
"run": {
"runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
},
"job": {
"namespace": "my-namespace",
"name": "my-job"
},
"inputs": [{
"namespace": "my-namespace",
"name": "my-input"
}],
"producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client"
}'

RESPONSE

201 CREATED

Step 2: Complete a Run

Use d46e465b-d358-4d32-83d4-df660ff614dd to complete the run for my-job with my-output as the output dataset. We also specify the schema facet to collect the schema for my-output before marking the run as completed. Note, you don't have to specify the input dataset my-input again for the run since it already has been associated with the run ID:

REQUEST

$ curl -X POST http://localhost:5000/api/v1/lineage \
-H 'Content-Type: application/json' \
-d '{
"eventType": "COMPLETE",
"eventTime": "2020-12-28T20:52:00.001+10:00",
"run": {
"runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
},
"job": {
"namespace": "my-namespace",
"name": "my-job"
},
"outputs": [{
"namespace": "my-namespace",
"name": "my-output",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet",
"fields": [
{ "name": "a", "type": "VARCHAR"},
{ "name": "b", "type": "VARCHAR"}
]
}
}
}],
"producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client"
}'

RESPONSE

201 CREATED

View Collected Lineage Metadata

Search Job Metadata

To view lineage metadata collected by Marquez, browse to the UI by visiting http://localhost:3000. Then, use the search bar in the upper right-side of the page and search for the job my-job. To view lineage metadata for my-job, click on the job from the drop-down list:

image

View Job Metadata

You should see the job namespace, name, my-input as an input dataset and my-output as an output dataset in the lineage graph and the job run marked as COMPLETED :

image

View Input Dataset Metadata

Finally, click on the output dataset my-output for my-job. You should see the dataset name, schema, and description:

image

Summary

In this simple example, we showed you how to use Marquez to collect dataset and job metadata with Openlineage. We also walked you through the set of HTTP API calls to successfully mark a run as complete and view the lineage metadata collected with Marquez.

Next Steps

Feedback

What did you think of this guide? We would love to hear feedback, and we can be found on the OpenLineage Slack.