0.13.0 - 2022-08-22
Added
- Add BigQuery check support
#960@denimalpaca
Adds logic and support for proper dynamic class inheritance for BigQuery-style operators. (BigQuery's extractor needed additional logic to support the forthcomingBigQueryColumnCheckOperatorandBigQueryTableCheckOperator.) - Add
RUNNINGEventTypein spec and Python client#972@mzareba382
Introduces aRUNNINGevent state in the OpenLineage spec to indicate a running task and adds aRUNNINGevent type in the Python API. - Use databases & schemas in SQL Extractors
#974@JDarDagran
Allows the Airflow integration to differentiate between databases and schemas. (There was no notion of databases and schemas when querying and parsing results frominformation_schematables.) - Implement Event forwarding feature via HTTP protocol
#995@howardyoo
AddsHttpLineageStreamto forward a given OpenLineage event to any HTTP endpoint. - Introduce
SymlinksDatasetFacetto spec#936@pawel-big-lebowski
Creates a new facet, theSymlinksDatasetFacet, to support the storing of alternative dataset names. - Add Azure Cosmos Handler to Spark integration
#983@hmoazam
Defines a new interface, theRelationHandler, to support Spark data sources that do not haveTableCatalog,Identifier, orTablePropertiesset, as is the case with the Azure Cosmos DB Spark connector. - Support OL Datasets in manual lineage inputs/outputs
#1015@conorbev
Allows Airflow users to create OpenLineage Dataset classes directly in DAGs with no conversion necessary. (Manual lineage definition required users to create anairflow.lineage.entities.Table, which was then converted to an OpenLineage Dataset.) - Create ownership facets
#996@julienledem
Adds an ownership facet to both Dataset and Job in the OpenLineage spec to capture ownership of jobs and datasets.
Changed
- Use
RUNNINGEventType in Flink integration for currently running jobs#985@mzareba382
Makes use of the newRUNNINGevent type in the Flink integration, changing events sent by Flink jobs fromOTHERto this new type. - Convert task objects to JSON-encodable objects when creating custom Airflow version facets
#1018@fm100
Implements ato_json_encodablefunction in the Airflow integration to make task objects JSON-encodable.
Fixed
- Add support for custom SQL queries in v3 Great Expectations API
#1025@collado-mike
Fixes support for custom SQL statements in the Great Expectations provider. (The Great Expectations custom SQL datasource was not applied to the support for the V3 checkpoints API.)