Skip to main content

v1.0.194

Date released: October 24, 2025

New features and enhancements​

The following new features are included in this release:

Backend​

Spot instance configuration for Spark ETL flows: This enhancement enables a new layer of control and cost optimization for running large-scale data transformations.

  • Smart resource allocation: Introduced a mechanism to automatically configure Spark jobs to utilize spot instances for executors and dedicated nodes for drivers within Kubernetes.

  • Cost optimization: This is achieved by mapping pipeline-specific or company-level node group selections (e.g., "medium," "large") to specific Kubernetes node selectors (e.g., mixedmedium) in the Spark configuration.

  • Benefit: Users can now seamlessly leverage cheaper compute resources for ETL workloads, resulting in significant cost savings without manually managing complex Kubernetes configurations.

Was this helpful?