1.1.0 - 2023-08-23
Added
- Flink: create Openlineage configuration based on Flink configuration
#2033@pawel-big-lebowski
Flink configuration entries starting withopenlineage.*are passed to the Openlineage client. - Java: add Javadocs to the Java client
#2004@julienledem
The client was missing some Javadocs. - Spark: append output dataset name to a job name
#2036@pawel-big-lebowski
Solves the problem of multiple jobs writing to different datasets while having the same job name. The feature is enabled by default and results in different job names. It can be disabled by settingspark.openlineage.jobName.appendDatasetNametofalse.
Unifies job names generated on the Databricks platform (using a dot job part separator instead of an underscore). The default behaviour can be altered withspark.openlineage.jobName.replaceDotWithUnderscore. - Spark: support Spark 3.4.1
#2057@pawel-big-lebowski
Bumps the latest Spark version to be covered in integration tests.
Fixed
- Airflow: do not use database as fallback when no schema parsed
#2023@mobuchowski
Sets the schema toNoneinTablesHierarchyto skip filtering on the schema level in the information schema query. - Flink: fix a bug when getting schema for
KafkaSink#2042@pentium3
Fixes the incomplete schema fromKafkaSinkVisitorby changing theKafkaSinkWrapperto catch schemas of typeAvroSerializationSchema. - Spark: filter
CreateViewevents#1968#1987@pawel-big-lebowski
Clears events generated by logical plans havingCreateViewnodes as root. - Spark: fix
MERGE INTOfor delta tables identified by physical locations#2026@pawel-big-lebowski
Delta tables identified by physical locations were not properly recognized. - Spark: fix incorrect naming of JDBC datasets
#2035@mobuchowski
Makes the namespace generated by the JDBC/Spark connector conform to the naming schema in the spec. - Spark: fix ignored event
adaptive_spark_planin Databricks#2061@algorithmy1
Removesadaptive_spark_planfrom theexcludedNodesinDatabricksEventFilter.