Skip to main content
Version: Next

1.36.0 - 2025-07-22

Added​

  • Spark: support Delta 4.0 and cover it with tests on Spark 4.0. #3877 @pawel-big-lebowski
    Fix failing tests for Spark 4.0. Make delta integration tests pass with Delta 4.0 on Spark 4.
  • Spark: Add memory info to debug facet. #3914 @pawel-big-lebowski
    Extend DebugFacet with additional information on Spark's driver memory configuration and current memory usage.
  • Spark: Add new AlterTableCommandDatasetBuilder for Spark 4.0. #3921 @pawel-big-lebowski
    Add support for AlterTableCommand dataset building in Spark 4.0.
  • dbt: Add query IDs for dbt. #3890 @jroachgolf84
    Add query ID tracking to dbt integration.
  • dbt: Add query ID capture in structured logs. #3918 @mobuchowski
    Capture query IDs from dbt structured logs for better traceability.
  • Python: Formalize dataset naming for Python client. #3816 @ddebowczyk92
    Formalize dataset naming conventions in Python client implementation.

Changed​

  • Spark: bump minor versions 3.4.3 -> 3.4.4, 3.5.4 -> 3.5.6. #3907 @pawel-big-lebowski
    Bump tested Spark versions.
  • Spark: Close OpenLineageClient in onApplicationEnd. #3851 @dolfinus
    Ensure proper cleanup of OpenLineageClient when Spark application ends.
  • Python: Do not use f-strings with logging module. #3895 @mobuchowski
    Replace f-string usage in logging calls with proper logging formatting.
  • Python: Update protobuf version to be compatible with newer libraries. #3899 @mobuchowski
    Update protobuf dependency to maintain compatibility with newer library versions.
  • Website: Documentation for compatibility tests. #3869 @mobuchowski
    Add documentation explaining compatibility testing processes.

Fixed​

  • Spark: make visitors stateless - avoid memory leak. #3902 @pawel-big-lebowski
    Merge SqlExecutionRDDVisitor and LogicalRDDVisitor classes to avoid memory leak.
  • Spark: refactor iceberg handler. #3909 @pawel-big-lebowski
    Refactor Iceberg handler implementation for better maintainability.
  • Spark: retry exception on empty row. #3908 @pawel-big-lebowski
    Add retry logic for handling empty row exceptions.
  • Spark: fix Spark version for databricks test. #3911 @pawel-big-lebowski
    Fix Spark version configuration in Databricks test environment.
  • Flink: Fix connector of type kafka-upsert not identifying kafka topics correctly. #3915 @fetta
    Fix kafka-upsert connector to properly identify kafka topics.
  • Airflow: Fail fast and reduce timeout for airflow tests. #3905 @kacpermuda
    Improve test performance by implementing fail-fast behavior and reduced timeouts.
  • dbt: more telemetry, fix quadratic file reading. #3916 @mobuchowski
    Improve telemetry collection and fix performance issues with file reading.
  • dbt: Fix dbt version. #3894 @mobuchowski
    Fix dbt version compatibility issues.
  • Python: Fix filenames for windows users. #3889 @pawel-big-lebowski
    Fix filename handling to work correctly on Windows systems.
  • Transport: Adjust log level when aliasing default_http transport. #3897 @dolfinus
    Adjust logging level for transport aliasing messages.
  • Build: Improve comments and add some tests. #3901 @kacpermuda
    Improve code documentation and add additional test coverage.