Skip to main content
Version: Next

1.37.0 - 2025-08-11

Added​

  • Spark: Implement support for WriteDelta, WriteIcebergDelta logical plan nodes #3860 @orthoxerox
    Add support for WriteDelta and WriteIcebergDelta logical plan nodes in Spark integration.
  • dbt: Add option to override dbt job name #3933 @mobuchowski
    Add configuration option to override dbt job names in OpenLineage events.
  • Java: Add Jackson Blackbird module for JSON performance optimization #3923 @kyungryun
    Improve JSON serialization performance with Jackson Blackbird module.
  • Python: Add Datadog transport #3950 @mobuchowski
    Add Datadog transport with routing between sync/async transports based on configurable rules. Provides easier integration with Datadog's data observability platform.

Changed​

  • Spark: Remove Spark 2 support #3904 @pawel-big-lebowski
    Drop support for Spark 2.x versions. Minimum supported version is now Spark 3.x.
  • Python: Change gzip compression level in HTTP transport #3956 @dolfinus
    Optimize HTTP transport performance by adjusting gzip compression level.
  • Spark: Add support for Spark 4 in streaming tests #3925 @SalvadorRomo
    Extend streaming integration tests to support Spark 4.0.

Fixed​

  • Spark: Improve performance of column level lineage #3946 @pawel-big-lebowski
    Limit memory consumption, provide limits for the amount of dependencies processed (1M) and input fields returned in the facet (100K). Turns on dataset lineage by default.
  • Spark: Add schema size limit for column level lineage processing #3949 @ddebowczyk92
    Add limits to prevent performance issues with large schemas in column-level lineage processing.
  • Spark: Fix context factory for Spark 4 #3934 @pawel-big-lebowski
    Fix context factory implementation for Spark 4.0 compatibility.
  • Spark: Fix LogicalRelation constructor compatibility for Spark 4 #3930 @yunchipang
    Fix LogicalRelation constructor to maintain compatibility with Spark 4.0.
  • Spark: Fix vendors parsing in SparkOpenLineageConfig #3947 @ddebowczyk92
    Fix parsing of vendor configurations in Spark OpenLineage configuration.
  • dbt: Use correct namespace for dbt externalQuery facet #3953 @jroachgolf84
    Fix namespace handling in dbt external query facets.
  • Python: Fix tags configuration #3943 @JDarDagran
    Fix configuration handling for user-supplied tags in Python client.