Build

Java 17

Testing requires a Java 17 JVM to test the Scala Spark components. Use your favourite tool (sdkman, /usr/libexec/java_home) to set JAVA_HOME and PATH environmental variables properly.

Preparation

The integration depends on four libraries that are build locally openlineage-java, spark-extension-interfaces, spark-extension-entrypoint and openlineage-sql-java, so before any testing or building of a package you need to publish the appropriate artifacts to local maven repository. To build the packages you need to execute.

To install openlineage-java in local maven repo run:

cd ../../client/java/ && ./gradlew publishToMavenLocal

For spark-extension-interfaces run:

cd ../../integration/spark-extension-interfaces && ./gradlew publishToMavenLocal

For spark-extension-entrypoint run:

cd ../../integration/spark-extension-entrypoint && ./gradlew publishToMavenLocal

For openlineage-sql-java run:

../../integration/sql/iface-java/ && ./script/compile.sh
../../integration/sql/iface-java/ && ./script/build.sh

Testing

To run the tests, from the current directory run:

./gradlew test

To run the integration tests, from the current directory run:

./gradlew integrationTest

Build jar

./gradlew shadowJar

Contributing

If contributing changes, additions or fixes to the Spark integration, please include the following header in any new .java files:

/* 
/* Copyright 2018-2024 contributors to the OpenLineage project
/* SPDX-License-Identifier: Apache-2.0 
*/

A Github Action checks for headers in new .java files when pull requests are opened.

Thank you for your contributions to the project!