v0.4.2

What's Changed

• Add 'do not disrupt' karpenter annotations to the driver and executor

v0.4.1

What's Changed

• Add annotations to the executor to tell karpenter not to evict or consolidate pods

v0.4.0

What's Changed

• use relative imports when loading code within the module

This allows multiple installations to happily coexist side-by-side without clobbering each other.

v0.3.3

What's Changed

• setting a CPU request no longer also sets a limit automatically
• if you need to set a CPU limit on a driver/executor, call set_executor_cores_limit() or set_driver_cores_limit() on the builder
• those methods can also be explicitly passed None to disable limits
• otherwise the value passed must be an int that is at least 1 and greater than the requested CPUs

Why the change

Most k8s clusters are running pods as Docker containers, and so CPU limits are implemented using the kernel's ability to schedule processes on the CPU. In our experience, this can cause instability and poor performance for some applications. Most importantly, it can also result in poor resource utilisation on the node. Without the limit, the pods can use as much CPU as is available on the node, which can make workloads perform better.

v0.3.2

This release includes improvements to unit tests and various bug fixes and improvements. It also changes the CustomizableSparkKubernetesOperator to include the standard Airflow macros when templating job arguments.