0.13.0 - 2022-08-22
Added
- Add BigQuery check support
#960
@denimalpaca
Adds logic and support for proper dynamic class inheritance for BigQuery-style operators. (BigQuery's extractor needed additional logic to support the forthcomingBigQueryColumnCheckOperator
andBigQueryTableCheckOperator
.) - Add
RUNNING
EventType
in spec and Python client#972
@mzareba382
Introduces aRUNNING
event state in the OpenLineage spec to indicate a running task and adds aRUNNING
event type in the Python API. - Use databases & schemas in SQL Extractors
#974
@JDarDagran
Allows the Airflow integration to differentiate between databases and schemas. (There was no notion of databases and schemas when querying and parsing results frominformation_schema
tables.) - Implement Event forwarding feature via HTTP protocol
#995
@howardyoo
AddsHttpLineageStream
to forward a given OpenLineage event to any HTTP endpoint. - Introduce
SymlinksDatasetFacet
to spec#936
@pawel-big-lebowski
Creates a new facet, theSymlinksDatasetFacet
, to support the storing of alternative dataset names. - Add Azure Cosmos Handler to Spark integration
#983
@hmoazam
Defines a new interface, theRelationHandler
, to support Spark data sources that do not haveTableCatalog
,Identifier
, orTableProperties
set, as is the case with the Azure Cosmos DB Spark connector. - Support OL Datasets in manual lineage inputs/outputs
#1015
@conorbev
Allows Airflow users to create OpenLineage Dataset classes directly in DAGs with no conversion necessary. (Manual lineage definition required users to create anairflow.lineage.entities.Table
, which was then converted to an OpenLineage Dataset.) - Create ownership facets
#996
@julienledem
Adds an ownership facet to both Dataset and Job in the OpenLineage spec to capture ownership of jobs and datasets.
Changed
- Use
RUNNING
EventType in Flink integration for currently running jobs#985
@mzareba382
Makes use of the newRUNNING
event type in the Flink integration, changing events sent by Flink jobs fromOTHER
to this new type. - Convert task objects to JSON-encodable objects when creating custom Airflow version facets
#1018
@fm100
Implements ato_json_encodable
function in the Airflow integration to make task objects JSON-encodable.
Fixed
- Add support for custom SQL queries in v3 Great Expectations API
#1025
@collado-mike
Fixes support for custom SQL statements in the Great Expectations provider. (The Great Expectations custom SQL datasource was not applied to the support for the V3 checkpoints API.)