Kubex Automation Engine

​

Breaking

• GPU enablement defaults and experimental contract: GPU request actions now default to disabled. spec.enablement.gpu.requests.downsize, .upsize, and .setFromUnspecified changed from true to false. The GPU/KAI experimental contract changed to v1alpha1-2026-07; the previous contract is no longer accepted. Affects AutomationStrategy, ClusterAutomationStrategy, GpuRebalancingPolicy, ClusterGpuRebalancingPolicy, and GpuConsolidationPolicy.
• After upgrading the CRDs, update spec.experimental.gpuKaiContract from v1alpha1-2026-04 to v1alpha1-2026-07 in every affected resource.
• For AutomationStrategy/ClusterAutomationStrategy, explicitly set each desired GPU action under spec.enablement.gpu.requests to true if you rely on GPU request resizing. GPU policy kinds only need the contract update.
• Reapply affected resources after making these changes.

• Rollback monitoring now reopens automatically when a newer replacement pod appears carrying the same recommendation and matching resources, so a pod recreated by eviction, in-place resize, or a manual restart resumes being tracked instead of starting a brand-new monitoring turn.
• RollbackPolicy/ClusterRollbackPolicy gain spec.enableMonitoringReopen (default true) to opt out of this reopen behavior for a given policy.
• Manifest-level resource sizing is now reported to the Kubex backend, giving visibility into a workload’s declared (manifest) resources alongside its live/adopted values.

• Secondary/DR cluster mode for recommendation consumption from a primary cluster.

• KubeAI Model workload support for automated rightsizing and rollback tracking.
• GPU-sharing tuning controls for KAI-based automation strategies.
• Container-level skip controls for rightsizing, so specific containers can be excluded without blocking resize actions for the rest of the pod.

• Default policy evaluation now includes KubeAI Model workloads when workloadTypes is not set.
• VPA-aware resize handling is more consistent across live workloads.

• Helm uninstall reliability for charts using rightsizing resources.
• GPU rightsizing and validation stability.
• In-place resized pods now retain the pod-rightsizing-info annotation so live pod state stays observable.

• KubeAI Model workload support, including owner-scoped recommendation and rollback state handling.
• KAI/vLLM tuning controls for GPU-sharing automation strategies.

• Default policy workload scope now includes KubeAI Model objects when workloadTypes is omitted.
• Rollback monitoring now enforces adoption-threshold behavior more consistently.
• Agentic Proposal sync remains opt-in through GlobalConfiguration and is disabled by default.

• Helm uninstall reliability by aligning the pre-delete cleanup hook permissions with the rightsizing resources it patches.

• Initial formal release of the integration with the KAI scheduler for rightsizing fractional GPUs

• KAI documentation, examples, and release notes were updated for the v1.4.0 release

• Default policy evaluation now gives RollbackPolicy and ClusterRollbackPolicy the highest precedence.

• Introduced Rollback automation so clusters can now use the rollback state machine in live environments.

• Rollback owner/runtime handling and e2e fixtures were updated to support the new rollback flow.

• Live rollback e2e instability caused by synthetic state seeding.

• Webhook/client behavior improvements that make policy admission and reconciliation more resilient.

• Webhook reconciliation now uses informers/client behavior tuned for more reliable event handling.
• GPU-related e2e and policy behavior was refined for stability and consistency.
• Chart/docs content was updated to reflect the current release flow and user-facing guidance.

• Webhook error handling paths that could surface avoidable failures.
• Miscellaneous release-blocking regressions from the beta cycle.

• PodAffinityPolicy automation for supported workload types, including preferred node placement rules
• StrimziPodSet workload support for automating Strimzi-managed Kafka pods
• Prometheus scraping support for controller metrics with a chart-managed metrics service and optional ServiceMonitor
• Added experimental support for GPU sharing using the KAI scheduler

• GPU proactive policies can use gpuOverallOptimal recommendations from KAI for overall GPU optimization
• Resize summaries now show when recommendations were clamped to configured resource bounds

• No customer-facing changes in this release.

• Namespace-level pause controls using the rightsizing.kubex.ai/pause-until annotation so automation can be paused across an entire namespace without annotating each pod individually

• Time-based automation scheduling with configurable windows including all-day support, overnight windows, and 24:00 explicit end time
• Policy snapshot uploads to Kubex gateway via automation-gateway with configurable intervals

• VPA resizing now requires active VPA recommendation conditions before allowing resize plans to prevent premature operations

• Stale owner recommendation cleanup flow for missing automation strategies
• Overlapping exclusion window handling by jumping to the latest end time
• Startup policy rescan now triggers automatically after readiness opens

• Configurable webhook probe pod settings via GlobalConfiguration including image selection
• ImagePullSecrets and securityContext support for enhanced security configuration
• GlobalConfiguration singleton enforcement via validating webhook to prevent multiple instances

• Case-insensitive HPA kind detection to match HPA targets regardless of casing
• Guaranteed QoS resize plan calculation to correctly normalize requests and limits

• Guaranteed QoS support with retainGuaranteedQOS flag to enforce requests==limits constraint

• Webhook retry handling to properly manage too-many-requests scenarios
• Resize method preservation in retry scenarios to maintain consistency across retries
• Workload-to-policy namespace matching to ensure correct policy application

• Per-container enablement bounds to set different resource floor and ceiling limits for individual containers within a pod

• Webhook validation for inherited automation strategy bounds to ensure proper constraint enforcement

• Webhook validation for policy automation strategy references to ensure referential integrity
• Protection against deleting automation strategies that are actively referenced by policies

• ENABLE_WEBHOOKS flag parsing to correctly interpret as boolean value

• Rebranded from previous name to Kubex

• Workload-to-policy namespace matching to ensure policies are applied to correct workloads

• Provided complete security contexts to resolve Helm coalesce.go:298 warnings for valkey.podSecurityContext and valkey.securityContext.
• Added initContainer resource requests/limits to satisfy OpenShift admission requirements.
• Removed fixed UIDs/GIDs from valkey for OpenShift SCC compatibility.

• Automatic VerticalPodAutoscaler (VPA) detection and exclusion to prevent Kubex Automation from competing with VPA for resource management.
• Prevents conflicts between VPA and Kubex automation.

• Removed OpenShift SCC rolebinding configuration-no longer requires anyuid for Kubex Automation Controller.

• Added OpenShift support for Kubex Automation Controller.
• Removed default valkey storage settings-now ephemeral by default.

• Human-readable timestamps for automation logs—All automation logs now use clear, human-readable timestamps, improving readability, troubleshooting, and auditability.
• Support for user-defined metadata on service accounts—Users can now attach additional, custom metadata to service accounts.

• Easier Installation—Reduced deployment complexity and external dependencies by no longer requiring cert-manager by default.
• Flexible Options—Continue to choose from self-signed certificates, cert-manager integration, or bring-your-own-certificate based on your environment requirements.

• Private Registry Support—Organizations that cannot pull images from public repositories can now configure the automation controller to use images hosted in their private registries.
• Image Location Override—The waitForValkeyImage configuration option allows specifying a custom image location (default: busybox:latest), enabling deployment in air-gapped or restricted network environments.
• Flexible Deployment—This enhancement ensures the automation controller can be deployed in security-hardened environments where all container images must be sourced from approved internal registries.

• Zero-Downtime Resizing—Containers are resized in-place without eviction when supported by the cluster, ensuring continuous availability for critical workloads.
• Automatic Fallback—For clusters that don’t support in-place resizing or workloads that require pod recreation, the controller automatically falls back to the traditional pod eviction method.
• Kubernetes 1.33+ Required—In-place resizing requires Kubernetes 1.33 or later with the InPlacePodVerticalScaling feature gate enabled.

• RESIZED—Pod was successfully resized using in-place resizing (zero downtime).
• EVICTED—Pod was evicted and recreated with new resource specifications (traditional method).
• BLOCKED—Pod could not be resized, with detailed blocking reasons explaining why (e.g., hpa_conflict_cpu, limit_range_violation, resource_quota_exceeded, node_size_insufficient, manual_pause_infinite).
• SKIPPED—Pod is already sized at the recommended specification, no action needed.

• Learning Periods—Pause automation temporarily after application changes or deployments to allow the system to gather sufficient metrics before optimizing. Specify an RFC3339 timestamp to pause until a specific date.
• Permanent Exclusions—Permanently exclude specific workloads from automation by setting the annotation value to infinite, providing granular control over which resources are managed by Kubex.

• Previously, podLabels was mandatory and required explicit configuration even when not needed.
• Now, you can define scope using only namespace for namespace-wide automation, or combine namespace with optional podLabels for more granular control.

• Quick Initial Scan—The first scan now executes 2 minutes after controller startup (default), allowing rapid initialization and faster time-to-value.
• Configurable Regular Interval—Subsequent scans run at the configured interval, providing predictable automation cadence.
• Reduced Startup Delay—Previously, users had to wait for the full scan interval before the first automation actions occurred.

• Increased API Timeout—API timeout values have been increased to handle slower Kubex API responses during peak loads, preventing unnecessary automation failures.
• Updated cert-manager Dependency—The cert-manager version has been updated to the latest stable release, ensuring compatibility with modern Kubernetes clusters and improving certificate management reliability.

• Reduced Webhook Invocations—The mutating admission controller now only intercepts pod creation events, eliminating unnecessary processing during pod updates.
• Improved Cluster Performance—Reducing webhook triggers decreases API server load and improves overall cluster responsiveness.

• Future Feature Preparation—RBAC rules have been added to grant the controller the necessary permissions to perform in-place pod resource updates when the feature becomes generally available.
• No Functional Changes—This release focuses on infrastructure readiness; the in-place resizing feature itself was introduced in version 1.0.7.

• External Secrets Integration—The Helm chart now supports working with secrets created and managed by external secret management tools (e.g., External Secrets Operator, Sealed Secrets), rather than requiring the chart to create secrets internally.
• Enhanced Security Posture—Organizations can now leverage their existing secret management workflows and tools, ensuring secrets are handled according to enterprise security policies and compliance requirements.
• Customer-Driven Enhancement—This feature was developed based on customer feedback to support real-world enterprise deployment scenarios where centralized secret management is required.

• Arbitrary Namespace Support—The Helm chart can now be deployed in any namespace, not just predefined ones, providing greater flexibility for multi-tenant environments and organizational policies.
• Improved Secret Management—Namespace flexibility complements external secret management by allowing secrets to reside in different namespaces according to security boundaries and access control requirements.
• Simplified Multi-Cluster Deployments—Organizations with standardized namespace naming conventions can now deploy the controller consistently across multiple clusters without chart modifications.

• Namespace Context—Environment variables now include the release namespace, enabling better multi-namespace deployments and troubleshooting.
• Improved Configuration—Simplifies configuration in multi-tenant environments where multiple instances of the controller run in different namespaces.

• Simplified Deployment—Eliminates the need for persistent storage configuration, reducing deployment complexity and storage management overhead.
• Improved Performance—Valkey provides fast in-memory data access for recommendation retrieval, improving controller response times.
• Enhanced Scalability—In-memory storage enables better horizontal scaling capabilities for large-scale deployments.

• Automated Pod Scanning—The controller continuously scans all pods within the configured scope, evaluating them for optimization opportunities based on Kubex recommendations.
• Pre-Eviction Safety Checks—Before resizing any pod, the controller runs a comprehensive series of checks to ensure safe eviction, including:
  • Validation of workload owner types and controller compatibility
  • HPA (Horizontal Pod Autoscaler) conflict detection
  • Resource quota and limit range compliance
  • Pod disruption budget (PDB) verification
  • Policy specification compliance to verify automation is allowed
• Intelligent Pod Eviction—When a pod passes all safety checks, the controller performs controlled eviction, allowing the mutating admission controller to automatically apply optimized resource specifications when the pod is recreated.
• Scope-Based Control—Administrators define automation scope through Helm values, specifying namespaces and optional pod labels to control which workloads are managed by the automation controller.

• Gateway Resource Management—Added resource requests and limits for the gateway container, ensuring predictable resource allocation and preventing resource contention in production environments.
• Enhanced Secret Security—Improved secret handling by switching from environment variables to volume mounts, reducing the risk of secret exposure through process listings and container inspection.
• Automated CA Bundle Management—The webhook configuration now automatically extracts the CA bundle from TLS secrets, eliminating the need for manual CA bundle configuration and reducing deployment complexity.
• Simplified Configuration—Removed the manual caBundle dependency from values-edit.yaml, streamlining the Helm chart configuration process.
• Improved Documentation—Enhanced certificate generation documentation with detailed examples for multiple certificate creation methods, including OpenSSL, CFSSL, and Bring Your Own Certificate (BYOC) scenarios.

• Helm Chart Deployment—Packaged as a Helm release for simplified installation and management.
• Automated Setup Script—Quick deployment script automates setup, including cert-manager installation.
• Split Architecture—Redesigned with separate containers for core webhook logic and secure API communication.

• Enhanced Security—Encrypted credentials, non-root container execution, and seccomp profiles enabled by default.
• Selective Automation—Ability to include or exclude specific workload types (e.g., Deployment, StatefulSet) from automation.
• Offline Resilience—Mutations are cached locally during connectivity issues and synchronized when connection is restored.