Package io.openlineage.client.circuitBreaker

Class TaskQueueCircuitBreaker

java.lang.Object
io.openlineage.client.circuitBreaker.TaskQueueCircuitBreaker
All Implemented Interfaces:
CircuitBreaker
public class TaskQueueCircuitBreaker extends Object implements CircuitBreaker
Queues every openlineage task for execution by a bounded threadpool to prevent creation of too many threads (unlike cachedthreadpool) and resulting impact on the spark job. Once queued, the circuit breaker waits for some configured time for the task to finish execution in an effort to preserve the ordering of completion of tasks. If queue is full, it gives up on the task. An explicit close() that need to be called after the application end event is processed shuts it down after waiting for a while (configurable) to finish pending tasks. Also the circuit breaker maintains the count of rejected, canceled tasks, or submission timeouts.