Skip to main content

Simplify OpenLineage Configuration with Dynamic Environment Variables

· 4 min read
Jakub Dardziński

The OpenLineage community has frequently highlighted the need for a more flexible and scalable approach to configuration management, particularly through environment variables. Many users reported challenges in maintaining separate configuration files across development, testing, and production environments. In response, the 1.23.0 release introduces dynamic environment variables—addressing these concerns head-on by simplifying the configuration process, reducing code changes, and improving security across different environments.

In this guide, we'll explore how to leverage dynamic environment variables to simplify OpenLineage configuration, enhance flexibility, and improve security.

What's New in 1.23.0

In the upcoming 1.23.0 release of OpenLineage, we're excited to introduce dynamic environment variables, which will revolutionize the way you configure your OpenLineage client. With this feature, you'll be able to simplify your configuration, improve maintainability, and adapt to changing environments with ease.

Why Dynamic Environment Variables?

Flexibility on a Whole New Level​

Stop worrying about reconfiguring code or files—dynamic environment variables let you adjust settings on the fly. This feature is perfect for developers, testers, and production environments that demand real-time adaptability.

Centralized Management at Your Fingertips​

With environment variables, you can securely store configuration details alongside other essential application settings. No more scattering information across multiple files—it’s all in one place.

Cleaner Code, Happier Developers​

Say goodbye to the clutter of parsing configuration files in your codebase. Dynamic environment variables lead to cleaner, more maintainable applications, saving you time and headaches.

Seamless Integration for Effortless Deployment​

Dynamic environment variables easily sync with modern deployment tools and practices, ensuring smooth and stress-free configuration management every time.

When to Use Dynamic Environment Variables​

Development & Testing​

Need to experiment with new configurations quickly? Dynamic environment variables let you do just that—without ever touching your code.

Production Deployments​

Keep your production environment secure and organized by managing all configuration settings with ease and efficiency.

Dynamic Environments​

No more manual updates! Easily adapt to rapidly changing environments with this flexible configuration approach.

Key Features You’ll Love​

  • Dynamic Updates: Modify configurations in real-time without restarting your application.
  • Hierarchical Organization: Keep your settings clear and manageable with nested sections.
  • Automatic Conversions: OpenLineage automatically converts variable names to camelCase in Java, ensuring consistency.
  • JSON Support: Got complex configurations? Embed them directly in environment variables with JSON.
  • Effortless Integration: Tie your configuration process into deployment tools and scripting languages for smooth automation.

Quick Setup Example: HTTP Transport Configuration​

Let’s see dynamic environment variables in action! Suppose you need to configure your client to send events to an HTTP endpoint with basic authentication. Here’s how simple it is using environment variables:

OPENLINEAGE__TRANSPORT__TYPE=http
OPENLINEAGE__TRANSPORT__URL=http://localhost:5050
OPENLINEAGE__TRANSPORT__ENDPOINT=/api/v1/lineage
OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key
OPENLINEAGE__TRANSPORT__AUTH__API_KEY=your_api_key
OPENLINEAGE__TRANSPORT__COMPRESSION=gzip

This translates to the equivalent YAML configuration:

transport:
  type: http
  url: http://localhost:5050
  endpoint: /api/v1/lineage
  auth:
    type: api_key
    apiKey: your_api_key
  compression: gzip

Configuring CompositeTransport with Aliases for Ultimate Flexibility​

In Python, configuring CompositeTransport is a breeze with aliases for OPENLINEAGE_URL, OPENLINEAGE_API_KEY, and OPENLINEAGE_ENDPOINT. It allows you to create a default configuration for the default HTTP transport and easily add extra targets.

Here’s an example if you’ve already set the following environment variables:

OPENLINEAGE_URL=http://localhost:5050
OPENLINEAGE_API_KEY=your_api_key
OPENLINEAGE_ENDPOINT=/api/v1/lineage

Now, configure the CompositeTransport:

OPENLINEAGE__TRANSPORT__TYPE=composite
OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__TYPE=http
OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__URL=http://another_host:5050
OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__AUTH='{"type":"api_key", "apiKey":"random_token"}'
OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__COMPRESSION=gzip

You’ve just configured a CompositeTransport with two targets: default_http and another_target. The best part? You didn’t have to touch your config files!

Troubleshooting​

Got issues? Here’s how to troubleshoot common problems with OpenLineage configuration:

  • Check Variable Names: Ensure the environment variable names match what OpenLineage expects.
  • Validate Values: Double-check that you’re assigning the correct values.
  • Client Permissions: Ensure your OpenLineage client has access to read the environment variables, especially in containers like Docker.
  • Environment Variable Verification: Confirm all necessary environment variables are set in the correct environment.

Wrap up: Embrace Dynamic Configuration​

Dynamic environment variables offer a powerful and flexible approach to configuring the OpenLineage client. By leveraging this feature, you can streamline your setup, improve maintainability, and adapt configurations seamlessly within your dynamic environments. Say goodbye to complex configuration files and hello to a more efficient and scalable OpenLineage setup!

Ready to learn more? Check out the OpenLineage documentation for more information on configuring the CompositeTransport and other advanced features.