Reusable actions and common scripts
Reusable actions
Run Event Validation
The run_event_validation
action is a custom GitHub action that handles validation logic for OpenLineage events. Because OpenLineage events have a standardized structure, we provide a generic action that validates events against OpenLineage specifications.
The action:
- Retrieves the OpenLineage specification for all releases defined in
release_tags
- Runs syntax validation (checks if events conform to the OpenLineage JSON schema)
- Runs semantic validation (compares actual event content with expected values using Event Comparison)
- Creates a comprehensive report using Report
Inputs:
Name | Description | Required | Default |
---|---|---|---|
release_tags | List of the spec versions to check against | false | "" |
ol_release | Release to run the validation with | false | "" |
component_release | Release of the component producing events | false | "" |
target-path | Path to save the report to | true | - |
event-directory | Directory containing the events to validate | true | - |
producer-dir | Directory with producer definitions | true | - |
component | Component name to use | true | - |
Outputs:
Name | Description |
---|---|
report_path | Path to generated report |
Structure
The action requires a specific directory structure for validation to work properly:
Event Directory Structure:
- Root event directory - Top-level directory containing scenario subdirectories
- Scenario subdirectories - One directory per test scenario
- Generated event files - Actual OpenLineage events produced by the component being tested
- File naming - Events should be named descriptively (e.g.,
job_start.json
,job_complete.json
) - Format - All files must be valid JSON containing OpenLineage events
- Scenario subdirectories - One directory per test scenario
Producer Directory Structure:
- Producer root - Main directory for the producer component
- Scenarios directory - Contains expected event definitions
- Scenario subdirectories - Mirror the structure of event directory
config.json
- Configuration file with test specifications and version constraintsevents/
- Directory containing expected OpenLineage event templates- Expected event files - Template events using Jinja functions for flexible validation
maintainers.json
- File listing scenario maintainersscenario.md
- Documentation describing the test scenario
- Scenario subdirectories - Mirror the structure of event directory
- Scenarios directory - Contains expected event definitions
Example Directory Layout:
event-directory/
├── scenario1/
│ ├── job_start.json # Generated events
│ └── job_complete.json
└── scenario2/
├── spark_read.json
└── spark_write.json
producer-dir/
├── scenarios/
│ ├── scenario1/
│ │ ├── config.json # Test configuration
│ │ ├── events/
│ │ │ ├── job_start.json # Expected event template