Skip to main content
Version: 1.21.1

1.2.2 - 2023-09-20

Added​

  • Spark: publish the ProcessingEngineRunFacet as part of the normal operation of the OpenLineageSparkEventListener #2089 @d-m-h
    Publishes the spec-defined ProcessEngineRunFacet alongside the custom SparkVersionFacet (for now). The SparkVersionFacet is deprecated and will be removed in a future release.
  • Spark: capture and emit spark.databricks.clusterUsageTags.clusterAllTags variable from databricks environment #2099 @Anirudh181001
    Adds spark.databricks.clusterUsageTags.clusterAllTags to the list of environment variables captured from databricks.

Fixed​

  • Common: support parsing dbt_project.yml without target-path #2106 @tatiana
    As of dbt v1.5, usage of target-path in the dbt_project.yml file has been deprecated, now preferring a CLI flag or env var. It will be removed in a future version. This allows users to run DbtLocalArtifactProcessor in dbt projects that do not declare target-path.
  • Proxy: fix Proxy chart #2091 @harels
    Includes the proper image to deploy in the helm chart.
  • Python: fix serde filtering #2044 @xli-1026
    Fixes the bug causing values in list objects to be filtered accidentally.
  • Python: use non-deprecated apiKey if loading it from env variables @2029 @mobuchowski
    Changes api_key to apiKey in create_token_provider.
  • Spark: Improve RDDs on S3 integration. #2039 @pawel-big-lebowski
    Prepares integration test to access S3, fixes input dataset duplicates and includes other minor fixes.
  • Flink: prevent sending running events after job completes #2075 @pawel-big-lebowski
    Flink checkpoint tracking thread was not getting stopped properly on job complete.
  • Spark & Flink: Unify dataset naming from URI objects #2083 @pawel-big-lebowski
    Makes sure Spark and Flink generate same dataset identifiers for the same datasets by having a single implementation to generate dataset namespace and name.
  • Spark: Databricks improvements #2076 @pawel-big-lebowski
    Filters unwanted events on databricks and adds an integration test to verify this. Adds integration tests to verify dataset naming on databricks runtime is correct when table location is specified. Adds integration test for wide transformation on delta tables.

Removed​

  • SQL: remove sqlparser dependency from iface-java and iface-py #2090 @JDarDagran
    Removes the dependency due to a breaking change in the latest release of the parser.