Skip to main content

0.14.0 - 2022-09-06

Added

  • Support ABFSS and Hadoop Logical Relation in Column-level lineage #1008 @wjohnson
    Introduces an extractDatasetIdentifier that uses similar logic to InsertIntoHadoopFsRelationVisitor to pull out the path on the HDFS compliant file system; tested on ABFSS and DBFS (Databricks FileSystem) to prove that lineage could be extracted using non-SQL commands.
  • Add Kusto relation visitor #939 @hmoazam
    Implements a KustoRelationVisitor to support lineage for Azure Kusto's Spark connector.
  • Add ColumnLevelLineage facet doc #1020 @julienledem
    Adds documentation for the Column-level lineage facet.
  • Include symlinks dataset facet #935 @pawel-big-lebowski
    Includes the recently introduced SymlinkDatasetFacet in generated OpenLineage events.
  • Add support for dbt 1.3 beta's metadata changes #1051 @mobuchowski
    Makes projects that are composed of only SQL models work on 1.3 beta (dbt 1.3 renamed the compiled_sql field to compiled_code to support Python models). Does not provide support for dbt's Python models.
  • Support Flink 1.15 #1009 @mzareba382
    Adds support for Flink 1.15.
  • Add Redshift dialect to the SQL integration #1066 @mobuchowski
    Adds support for Redshift's SQL dialect in OpenLineage's SQL parser, including quirks such as the use of square brackets in JSON paths. (Note, this does not add support for all of Redshift's custom syntax.)

Changed

  • Make the timeout configurable in the Spark integration #1050 @tnazarew
    Makes timeout configurable by the user. (In some cases, the time needed to send events was longer than 5 seconds, which exceeded the timeout value.)

Fixed

  • Add a dialect parameter to Great Expectations SQL parser calls #1049 @collado-mike
    Specifies the dialect name from the SQL engine.
  • Fix Delta 2.1.0 with Spark 3.3.0 #1065 @pawel-big-lebowski
    Allows delta support for Spark 3.3 and fixes potential issues. (The Openlineage integration for Spark 3.3 was turned on without delta support, as delta did not support Spark 3.3 at that time.)