Skip to main content
Version: 1.32.1

1.28.0 - 2025-02-07

Added​

  • Java: enable specifying custom SSL context #3444 @pawel-big-lebowski Enable providing configuration for SSL context within HTTP transport.
  • Spark: make Spark nodes filtering configurable. #3442 @pawel-big-lebowski Spark integration filters OpenLineage events for specific plan node classes. This can be now extended with extra config entries: allowedSparkNodes and deniedSparkNodes. See Spark Configuration documentation for more details.
  • Java: add task queue based async circuit breaker. #3437 @aritrabandyo This circuit breaker that executes task on a queue backed threadpool, gives up tasks if the queue is full, and keeps track of rejected tasks.
  • dbt: added initial support for Trino adapter #3429 @whitleykeith This allows Trino integration to emit proper events containing Trino datasets.
  • Spark: increased coverage for Spark DML commands #3430 @ssanthanam185 Adds coverage for AlterTableRecoverPartitionsCommandVisitor, RefreshTableCommandVisitor, RepairTableCommandVisitor.

Changed​

  • Spark: the OpenLineageSparkListener was refactored to have a public, single-argument constructor taking an instance of SparkConf. #3425 @d-m-h This presents no functional change to the listener, however it will allow for improved initialisation of the listener in the future.
  • Spark: Unsupported catalog exception should be less verbose. #3435 @pawel-big-lebowski In case of unsupported classes, warn logs without a stacktrace should be produced.
  • Spark: Directly expose the LogicalPlan and SparkPlan objects inside OpenLineageContext. #3443 @d-m-h This is an initial refactor to a larger code base change that will see the removal of direct access of the QueryExecution object. It has no functional change on the way the integration behaves.

Fixed​

  • Spark: improve column lineage by including inputs within COMPLETE events. #3434 @pawel-big-lebowski *Send input datasets in COMPLETE events while making sure version facet is attached on START only.
  • dbt: ParentRunFacet is now correctly attached when using structured logs option. #3432 @MassyB Fixes incorrect structure of ParentRunFacet.