Skip to main content
Version: Next

1.23.0 - 2024-10-04

Added​

  • Java: added CompositeTransport #3039 @JDarDagran
    This allows user to specify multiple targets to which OpenLineage events will be emitted.
  • Spark extension interfaces: support table extended sources #3062 @Imbruced
    Interfaces are now able to extract lineage from Table interface, not only RelationProvider.
  • Java: added GCP Dataplex transport #3043 @ddebowczyk92
    Dataplex transport is now available as a separate Maven package for users that want to send OL events to GCP Dataplex.
  • Java: added Google Cloud Storage transport #3077 @ddebowczyk92
    GCS transport is now available as a separate Maven package for users that want to send OL events to Google Cloud Storage.
  • Java: added S3 transport #3129 @arturowczarek
    S3 transport is now available as a separate Maven package for users that want to send OL events to S3.
  • Java: add option to configure client via environment variables #3094 @JDarDagran
    Specified variables are now autotranslated to configuration values.
  • Python: add option to configure client via environment variables #3114 @JDarDagran
    Specified variables are now autotranslated to configuration values.
  • Python: add option to add custom headers in HTTP transport #3116 @JDarDagran
    Allows user to add custom headers, for example for auth purposes.
  • Spec: add full dataset dependencies #3097 #3098 @arturowczarek
    Now, if datasetLineageEnabled is enabled, and when column level lineage depends on the whole dataset, it does add dataset dependency instead of listing all the column fields in that dataset.
  • Java: OpenLineageClient and Transports are now AutoCloseable #3122 @ddebowczyk92
    This prevents a number of issues that might be caused by not closing underlying transports.

Fixed​

  • Python Facet generator does not validate optional arguments #3054 @JDarDagran
    This fixes issue where NominalTimeRunFacet Facet breaks when nominalEndTime is None.
  • SQL: report only actually used tables from CTEs, rather than all #2962 @Imbruced
    With this change, if SQL specified CTE, but does not use it in final query, the lineage won't be falsely reported.
  • Fluentd: Enhancing plugin's capabilities #3068 @jonathanlbt1
    This change enhances performance and docs of fluentd proxy plugin.
  • SQL: fix parser to point to origin table instead of CTEs #3107 @Imbruced
    For some complex CTEs, parser emitted CTE as a target table instead of original table. This is now fixed.
  • Spark: column lineage correctly produces for merge into command #3095 @Imbruced
    Now OL produces CLL correctly for the potential view in the middle.