0.13.0 - 2022-08-22
- Add BigQuery check support
Adds logic and support for proper dynamic class inheritance for BigQuery-style operators. (BigQuery's extractor needed additional logic to support the forthcoming
EventTypein spec and Python client
RUNNINGevent state in the OpenLineage spec to indicate a running task and adds a
RUNNINGevent type in the Python API.
- Use databases & schemas in SQL Extractors
Allows the Airflow integration to differentiate between databases and schemas. (There was no notion of databases and schemas when querying and parsing results from
- Implement Event forwarding feature via HTTP protocol
HttpLineageStreamto forward a given OpenLineage event to any HTTP endpoint.
Creates a new facet, the
SymlinksDatasetFacet, to support the storing of alternative dataset names.
- Add Azure Cosmos Handler to Spark integration
Defines a new interface, the
RelationHandler, to support Spark data sources that do not have
TablePropertiesset, as is the case with the Azure Cosmos DB Spark connector.
- Support OL Datasets in manual lineage inputs/outputs
Allows Airflow users to create OpenLineage Dataset classes directly in DAGs with no conversion necessary. (Manual lineage definition required users to create an
airflow.lineage.entities.Table, which was then converted to an OpenLineage Dataset.)
- Create ownership facets
Adds an ownership facet to both Dataset and Job in the OpenLineage spec to capture ownership of jobs and datasets.
RUNNINGEventType in Flink integration for currently running jobs
Makes use of the new
RUNNINGevent type in the Flink integration, changing events sent by Flink jobs from
OTHERto this new type.
- Convert task objects to JSON-encodable objects when creating custom Airflow version facets
to_json_encodablefunction in the Airflow integration to make task objects JSON-encodable.
- Add support for custom SQL queries in v3 Great Expectations API
Fixes support for custom SQL statements in the Great Expectations provider. (The Great Expectations custom SQL datasource was not applied to the support for the V3 checkpoints API.)