Skip to main content

0.16.1 - 2022-11-03

Added

  • Airflow: add dag_run information to Airflow version run facet #1133 @fm100
    Adds the Airflow DAG run ID to the taskInfo facet, making this additional information available to the integration.
  • Airflow: add LoggingMixin to extractors #1149 @JDarDagran
    Adds a LoggingMixin class to the custom extractor to make the output consistent with general Airflow and OpenLineage logging settings.
  • Airflow: add default extractor #1162 @mobuchowski
    Adds a DefaultExtractor to support the default implementation of OpenLineage for external operators without the need for custom extractors.
  • Airflow: add on_complete argument in DefaultExtractor #1188 @JDarDagran
    Adds support for running another method on extract_on_complete.
  • SQL: reorganize the library into multiple packages #1167 @StarostaGit @mobuchowski
    Splits the SQL library into a Rust implementation and foreign language bindings, easing the process of adding language interfaces. Also contains CI fix.

Changed

  • Airflow: move get_connection_uri as extractor's classmethod #1169 @JDarDagran
    The get_connection_uri method allowed for too many params, resulting in unnecessarily long URIs. This changes the logic to whitelisting per extractor.
  • Airflow: change get_openlineage_facets_on_start/complete behavior #1201 @JDarDagran
    Splits up the method for greater legibility and easier maintenance.

Fixed

  • Airflow: always send SQL in SqlJobFacet as a string #1143 @mobuchowski
    Changes the data type of query from array to string to an fix error in the RedshiftSQLOperator.
  • Airflow: include __extra__ case when filtering URI query params #1144 @JDarDagran
    Includes the conn.EXTRA_KEY in the get_connection_uri method to avoid exposing secrets in URIs via the __extra__ key.
  • Airflow: enforce column casing in SQLCheckExtractors #1159 @denimalpaca
    Uses the parent extractor's _is_uppercase_names property to determine if the column should be upper cased in the SQLColumnCheckExtractor's _get_input_facets() method.
  • Spark: prevent exception when no schema provided #1180 @pawel-big-lebowski
    Prevents evalution of column lineage when the schemFacet is null.
  • Great Expectations: add V3 API compatibility #1194 @denimalpaca
    Fixes the Pandas datasource to make it V3 API-compatible.

Removed

  • Airflow: remove support for Airflow 1.10 #1128 @mobuchowski
    Removes the code structures and tests enabling support for Airflow 1.10.