Skip to main content

Join us in London on January 31st

· One min read
Michael Robinson
OpenLineage Community Manager

Join us on Wednesday, January 31st, 2024, from 6:00-8:00 pm at the Confluent offices in London to learn more about the current state of lineage in general and streaming support in particular. Bring your ideas and vision for OpenLineage!

Meet Us in Warsaw on November 29th!

· One min read
Michael Robinson
OpenLineage Community Manager

Join us on Wednesday, November 29th, 2023, from 17:30-20:30 CET in Warsaw, Poland, to contribute to a discussion of the future of OpenLineage. On the tentative agenda:

  1. Mary Idamkina on OpenLineage in GCP Dataplex
  2. Paweł Leszczynski on recent developments in the Spark Integration
  3. Jakub Dardziński on Extracting lineage from PythonOperator - how come this is possible?
  4. Paweł Leszczynski on How to Become a Spark-OpenLineage Contributor in 5 Steps

Metaphor's Integration with OpenLineage Enhances Data Governance and Collaboration

· 3 min read
Yi Wang
Guest Blogger & Founding Engineer at Metaphor
Mars Lan
Guest Blogger & Co-founder/CTO at Metaphor

In the ever-evolving landscape of data management and governance, organizations constantly seek innovative solutions to streamline their processes, foster collaboration, and maximize the value of their data assets. Metaphor, born out of the minds behind LinkedIn's DataHub, has emerged as a modern data catalog and social platform for data. We take a unique approach by combining technical metadata with social collaboration, making data governance accessible and engaging for everyone in the organization. In this blog post, we explain the motivation behind Metaphor’s adoption of OpenLineage, delve into the integration methodology, and discuss its current status and benefits.

The OpenLineage Airflow Provider is Here

· 5 min read
Michael Robinson
OpenLineage Community Manager
Maciej Obuchowski
OpenLineage Committer
Julien Le Dem
OpenLineage Project Lead

This one is big. With the release of Airflow 2.7.0, the Airflow integration is now officially an Airflow Provider. This means the openlineage-airflow package is now apache-airflow-providers-openlineage in Airflow itself – a built-in feature of Airflow rather than an externally managed integration. Why does it matter where the integration “lives”? The short answer: as an Airflow Provider, the integration will offer improved reliability, broader support for operators, enhanced lineage, and easier implementation in custom operators going forward.

Although still a work in progress in some key respects, the OpenLineage Provider promises to pay dividends to users and contributors alike while accelerating the growth of the OpenLineage Ecosystem.

Meet Us in Toronto on September 18th!

· 2 min read
Michael Robinson
OpenLineage Community Manager

Join us on Monday, September 18th, 2023, from 5:00-8:00 pm PT ET in Toronto to contribute to a discussion of the future of OpenLineage. On the tentative agenda:

  • Intros
  • Evolution of spec presentation/discussion (project background/history)
  • State of the community
  • Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
  • Spark/Column lineage update
  • Airflow Provider update
  • Roadmap Discussion
  • Action items review/next steps

Bring your ideas and vision for OpenLineage!

Meet Us in San Francisco on August 30th!

· 2 min read
Michael Robinson
OpenLineage Community Manager

Join us on Wednesday, August 30th, 2023, from 5:30-8:30 pm PT at the Astronomer offices in San Francisco to learn more about the present and future of OpenLineage. Meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a discussion about the future of the project. Bring your ideas and vision for OpenLineage!

Also on the agenda: a presentation by new contributor/partner John Lukenoff, who will be speaking about his lineage use case.

Join us in New York on June 22nd

· One min read
Michael Robinson
OpenLineage Community Manager

Join us on Thursday, June 22nd, 2023, from 6:00-8:00 pm ET at Collibra's HQ in New York to discuss the present and future of OpenLineage. Meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a discussion about the future of the project. Bring your ideas and vision for OpenLineage!