> ## Documentation Index > Fetch the complete documentation index at: https://docs.kubex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Kubex Automation Engine ### Breaking * **[GPU enablement defaults and experimental contract](./BREAKING.md#2026-07-20---gpu-enablement-defaults-and-experimental-contract)**: GPU request actions now default to disabled. `spec.enablement.gpu.requests.downsize`, `.upsize`, and `.setFromUnspecified` changed from `true` to `false`. The GPU/KAI experimental contract changed to `v1alpha1-2026-07`; the previous contract is no longer accepted. Affects `AutomationStrategy`, `ClusterAutomationStrategy`, `GpuRebalancingPolicy`, `ClusterGpuRebalancingPolicy`, and `GpuConsolidationPolicy`. * After upgrading the CRDs, update `spec.experimental.gpuKaiContract` from `v1alpha1-2026-04` to `v1alpha1-2026-07` in every affected resource. * For `AutomationStrategy`/`ClusterAutomationStrategy`, explicitly set each desired GPU action under `spec.enablement.gpu.requests` to `true` if you rely on GPU request resizing. GPU policy kinds only need the contract update. * Reapply affected resources after making these changes. **Added:** * Rollback monitoring now reopens automatically when a newer replacement pod appears carrying the same recommendation and matching resources, so a pod recreated by eviction, in-place resize, or a manual restart resumes being tracked instead of starting a brand-new monitoring turn. * `RollbackPolicy`/`ClusterRollbackPolicy` gain `spec.enableMonitoringReopen` (default `true`) to opt out of this reopen behavior for a given policy. * Manifest-level resource sizing is now reported to the Kubex backend, giving visibility into a workload's declared (manifest) resources alongside its live/adopted values. **Added:** * Secondary/DR cluster mode for recommendation consumption from a primary cluster. **Added:** * KubeAI `Model` workload support for automated rightsizing and rollback tracking. * GPU-sharing tuning controls for KAI-based automation strategies. * Container-level skip controls for rightsizing, so specific containers can be excluded without blocking resize actions for the rest of the pod. **Changed:** * Default policy evaluation now includes KubeAI `Model` workloads when `workloadTypes` is not set. * VPA-aware resize handling is more consistent across live workloads. **Fixed:** * Helm uninstall reliability for charts using rightsizing resources. * GPU rightsizing and validation stability. * In-place resized pods now retain the pod-rightsizing-info annotation so live pod state stays observable. **Added:** * KubeAI `Model` workload support, including owner-scoped recommendation and rollback state handling. * KAI/vLLM tuning controls for GPU-sharing automation strategies. **Changed:** * Default policy workload scope now includes KubeAI `Model` objects when `workloadTypes` is omitted. * Rollback monitoring now enforces adoption-threshold behavior more consistently. * Agentic Proposal sync remains opt-in through `GlobalConfiguration` and is disabled by default. **Fixed:** * Helm uninstall reliability by aligning the pre-delete cleanup hook permissions with the rightsizing resources it patches. **Added:** * Initial formal release of the integration with the KAI scheduler for rightsizing fractional GPUs **Changed:** * KAI documentation, examples, and release notes were updated for the `v1.4.0` release **Changed:** * Default policy evaluation now gives `RollbackPolicy` and `ClusterRollbackPolicy` the highest precedence. **Added:** * Introduced Rollback automation so clusters can now use the rollback state machine in live environments. **Changed:** * Rollback owner/runtime handling and e2e fixtures were updated to support the new rollback flow. **Fixed:** * Live rollback e2e instability caused by synthetic state seeding. **Added:** * Webhook/client behavior improvements that make policy admission and reconciliation more resilient. **Changed:** * Webhook reconciliation now uses informers/client behavior tuned for more reliable event handling. * GPU-related e2e and policy behavior was refined for stability and consistency. * Chart/docs content was updated to reflect the current release flow and user-facing guidance. **Fixed:** * Webhook error handling paths that could surface avoidable failures. * Miscellaneous release-blocking regressions from the beta cycle. **Added:** * PodAffinityPolicy automation for supported workload types, including preferred node placement rules * StrimziPodSet workload support for automating Strimzi-managed Kafka pods * Prometheus scraping support for controller metrics with a chart-managed metrics service and optional ServiceMonitor * Added experimental support for GPU sharing using the KAI scheduler **Changed:** * GPU proactive policies can use `gpuOverallOptimal` recommendations from KAI for overall GPU optimization * Resize summaries now show when recommendations were clamped to configured resource bounds **Changed:** * No customer-facing changes in this release. **Added:** * Namespace-level pause controls using the `rightsizing.kubex.ai/pause-until` annotation so automation can be paused across an entire namespace without annotating each pod individually **Added:** * Time-based automation scheduling with configurable windows including all-day support, overnight windows, and 24:00 explicit end time * Policy snapshot uploads to Kubex gateway via automation-gateway with configurable intervals **Changed:** * VPA resizing now requires active VPA recommendation conditions before allowing resize plans to prevent premature operations **Fixed:** * Stale owner recommendation cleanup flow for missing automation strategies * Overlapping exclusion window handling by jumping to the latest end time * Startup policy rescan now triggers automatically after readiness opens **Added:** * Configurable webhook probe pod settings via GlobalConfiguration including image selection * ImagePullSecrets and securityContext support for enhanced security configuration * GlobalConfiguration singleton enforcement via validating webhook to prevent multiple instances **Fixed:** * Case-insensitive HPA kind detection to match HPA targets regardless of casing * Guaranteed QoS resize plan calculation to correctly normalize requests and limits **Added:** * Guaranteed QoS support with `retainGuaranteedQOS` flag to enforce requests==limits constraint **Fixed:** * Webhook retry handling to properly manage too-many-requests scenarios * Resize method preservation in retry scenarios to maintain consistency across retries * Workload-to-policy namespace matching to ensure correct policy application **Added:** * Per-container enablement bounds to set different resource floor and ceiling limits for individual containers within a pod **Fixed:** * Webhook validation for inherited automation strategy bounds to ensure proper constraint enforcement **Added:** * Webhook validation for policy automation strategy references to ensure referential integrity * Protection against deleting automation strategies that are actively referenced by policies **Fixed:** * ENABLE\_WEBHOOKS flag parsing to correctly interpret as boolean value **Changed:** * Rebranded from previous name to Kubex **Fixed:** * Workload-to-policy namespace matching to ensure policies are applied to correct workloads ### Legacy Kubex Automation Controller Release Notes The Kubex Automation Controller has been deprecated and replaced by the Kubex Automation Engine. The Kubex Automation Controller automatically implemented Kubex's optimization recommendations for Kubernetes workloads. The following improvements have been made in this release: * **Human-readable timestamps for automation logs**--All automation logs now use clear, human-readable timestamps, improving readability, troubleshooting, and auditability. * **Support for user-defined metadata on service accounts**--Users can now attach additional, custom metadata to service accounts. Streamlined deployment by changing the default certificate method, making cert-manager optional rather than required. * **Easier Installation**--Reduced deployment complexity and external dependencies by no longer requiring cert-manager by default. * **Flexible Options**--Continue to choose from self-signed certificates, cert-manager integration, or bring-your-own-certificate based on your environment requirements. Optimized default settings for Kubernetes 1.33+ clusters to take advantage of in-place resizing capabilities, significantly reducing time-to-optimization. In-place resizing operations are now visible in the Kubex UI, providing better insight into automation activities and outcomes. Added support for overriding the wait-for-Valkey init container image to accommodate environments with restricted access to public container registries. * **Private Registry Support**--Organizations that cannot pull images from public repositories can now configure the automation controller to use images hosted in their private registries. * **Image Location Override**--The `waitForValkeyImage` configuration option allows specifying a custom image location (default: `busybox:latest`), enabling deployment in air-gapped or restricted network environments. * **Flexible Deployment**--This enhancement ensures the automation controller can be deployed in security-hardened environments where all container images must be sourced from approved internal registries. The Kubex Automation Controller now supports Kubernetes In-Place Pod Resizing for eligible workloads running on Kubernetes 1.33 or higher. This capability enables resource adjustments without pod restarts, eliminating downtime during optimization. * **Zero-Downtime Resizing**--Containers are resized in-place without eviction when supported by the cluster, ensuring continuous availability for critical workloads. * **Automatic Fallback**--For clusters that don't support in-place resizing or workloads that require pod recreation, the controller automatically falls back to the traditional pod eviction method. * **Kubernetes 1.33+ Required**--In-place resizing requires Kubernetes 1.33 or later with the InPlacePodVerticalScaling feature gate enabled. Pod scan logs have been significantly improved to provide clear summaries of the actions taken by the automation controller. Each log entry now clearly indicates the action taken and the reason: * **RESIZED**--Pod was successfully resized using in-place resizing (zero downtime). * **EVICTED**--Pod was evicted and recreated with new resource specifications (traditional method). * **BLOCKED**--Pod could not be resized, with detailed blocking reasons explaining why (e.g., `hpa_conflict_cpu`, `limit_range_violation`, `resource_quota_exceeded`, `node_size_insufficient`, `manual_pause_infinite`). * **SKIPPED**--Pod is already sized at the recommended specification, no action needed. These enhanced logs make it easier to monitor automation activity, troubleshoot issues, and understand why specific pods were not optimized. Administrators can now use Kubernetes annotations to control automation behavior at the pod or workload level using the `rightsizing.kubex.ai/pause-until: ""` annotation: * **Learning Periods**--Pause automation temporarily after application changes or deployments to allow the system to gather sufficient metrics before optimizing. Specify an RFC3339 timestamp to pause until a specific date. * **Permanent Exclusions**--Permanently exclude specific workloads from automation by setting the annotation value to `infinite`, providing granular control over which resources are managed by Kubex. Additionally, the controller now respects the `cluster-autoscaler.kubernetes.io/safe-to-evict=false` annotation, ensuring that pods marked as unsafe to evict are never evicted by the automation controller, even when optimization recommendations are available. This annotation-based approach integrates seamlessly with GitOps workflows and provides declarative control over automation scope. See [Pausing Automation for Specific Pods](https://github.com/densify-dev/helm-charts/blob/master/charts/kubex-automation-controller/docs/Advanced-Configuration.md#pausing-automation-for-specific-pods) for detailed usage examples. The `podLabels` field in the Helm values scope configuration is now optional, simplifying deployment for users who want to enable automation across entire namespaces without label-based filtering. * Previously, `podLabels` was mandatory and required explicit configuration even when not needed. * Now, you can define scope using only `namespace` for namespace-wide automation, or combine `namespace` with optional `podLabels` for more granular control. The controller's pod scanning behavior has been enhanced to balance responsiveness with efficiency: * **Quick Initial Scan**--The first scan now executes 2 minutes after controller startup (default), allowing rapid initialization and faster time-to-value. * **Configurable Regular Interval**--Subsequent scans run at the configured interval, providing predictable automation cadence. * **Reduced Startup Delay**--Previously, users had to wait for the full scan interval before the first automation actions occurred. These improvements ensure new deployments are evaluated quickly while maintaining efficient resource utilization for ongoing operations. See [Pod Scan Configuration](https://github.com/densify-dev/helm-charts/blob/master/charts/kubex-automation-controller/docs/Pod-Scan-Configuration.md) for guidance on configuring scan intervals based on automation scope and cluster size. * **Increased API Timeout**--API timeout values have been increased to handle slower Kubex API responses during peak loads, preventing unnecessary automation failures. * **Updated cert-manager Dependency**--The cert-manager version has been updated to the latest stable release, ensuring compatibility with modern Kubernetes clusters and improving certificate management reliability. Optimized the mutating webhook configuration by removing the UPDATE operation, ensuring the webhook only triggers on pod creation events. * **Reduced Webhook Invocations**--The mutating admission controller now only intercepts pod creation events, eliminating unnecessary processing during pod updates. * **Improved Cluster Performance**--Reducing webhook triggers decreases API server load and improves overall cluster responsiveness. Added Kubernetes RBAC (Role-Based Access Control) permissions required for in-place pod resizing functionality. This update prepares the controller for the in-place resizing feature, which was still in development at the time of this release. * **Future Feature Preparation**--RBAC rules have been added to grant the controller the necessary permissions to perform in-place pod resource updates when the feature becomes generally available. * **No Functional Changes**--This release focuses on infrastructure readiness; the in-place resizing feature itself was introduced in version 1.0.7. Added support for external secret management tools, providing greater flexibility and security for enterprise deployments. * **External Secrets Integration**--The Helm chart now supports working with secrets created and managed by external secret management tools (e.g., External Secrets Operator, Sealed Secrets), rather than requiring the chart to create secrets internally. * **Enhanced Security Posture**--Organizations can now leverage their existing secret management workflows and tools, ensuring secrets are handled according to enterprise security policies and compliance requirements. * **Customer-Driven Enhancement**--This feature was developed based on customer feedback to support real-world enterprise deployment scenarios where centralized secret management is required. Removed namespace restrictions, allowing the Kubex Automation Controller to be deployed in any namespace. * **Arbitrary Namespace Support**--The Helm chart can now be deployed in any namespace, not just predefined ones, providing greater flexibility for multi-tenant environments and organizational policies. * **Improved Secret Management**--Namespace flexibility complements external secret management by allowing secrets to reside in different namespaces according to security boundaries and access control requirements. * **Simplified Multi-Cluster Deployments**--Organizations with standardized namespace naming conventions can now deploy the controller consistently across multiple clusters without chart modifications. Added release namespace information to relevant environment variables, improving namespace-aware operations. * **Namespace Context**--Environment variables now include the release namespace, enabling better multi-namespace deployments and troubleshooting. * **Improved Configuration**--Simplifies configuration in multi-tenant environments where multiple instances of the controller run in different namespaces. Replaced the persistent volume claim (PVC) requirement with Valkey (Redis-compatible) for storing optimization recommendations, simplifying deployment and improving scalability. * **Simplified Deployment**--Eliminates the need for persistent storage configuration, reducing deployment complexity and storage management overhead. * **Improved Performance**--Valkey provides fast in-memory data access for recommendation retrieval, improving controller response times. * **Enhanced Scalability**--In-memory storage enables better horizontal scaling capabilities for large-scale deployments. This release introduces the core Kubex Automation Controller deployment, which automatically manages container resource optimization through intelligent pod scanning and eviction. * **Automated Pod Scanning**--The controller continuously scans all pods within the configured scope, evaluating them for optimization opportunities based on Kubex recommendations. * **Pre-Eviction Safety Checks**--Before resizing any pod, the controller runs a comprehensive series of checks to ensure safe eviction, including: * Validation of workload owner types and controller compatibility * HPA (Horizontal Pod Autoscaler) conflict detection * Resource quota and limit range compliance * Pod disruption budget (PDB) verification * Policy specification compliance to verify automation is allowed * **Intelligent Pod Eviction**--When a pod passes all safety checks, the controller performs controlled eviction, allowing the mutating admission controller to automatically apply optimized resource specifications when the pod is recreated. * **Scope-Based Control**--Administrators define automation scope through Helm values, specifying namespaces and optional pod labels to control which workloads are managed by the automation controller. This release focuses on security enhancements and infrastructure improvements for the Kubex Automation Controller deployment. * **Gateway Resource Management**--Added resource requests and limits for the gateway container, ensuring predictable resource allocation and preventing resource contention in production environments. * **Enhanced Secret Security**--Improved secret handling by switching from environment variables to volume mounts, reducing the risk of secret exposure through process listings and container inspection. * **Automated CA Bundle Management**--The webhook configuration now automatically extracts the CA bundle from TLS secrets, eliminating the need for manual CA bundle configuration and reducing deployment complexity. * **Simplified Configuration**--Removed the manual `caBundle` dependency from `values-edit.yaml`, streamlining the Helm chart configuration process. * **Improved Documentation**--Enhanced certificate generation documentation with detailed examples for multiple certificate creation methods, including OpenSSL, CFSSL, and Bring Your Own Certificate (BYOC) scenarios. This inaugural release introduces the Kubex Automation Controller, rebranded from the Mutating Admission Controller to reflect broader automation capabilities. * **Helm Chart Deployment**--Packaged as a Helm release for simplified installation and management. * **Automated Setup Script**--Quick deployment script automates setup, including cert-manager installation. * **Split Architecture**--Redesigned with separate containers for core webhook logic and secure API communication. * **Enhanced Security**--Encrypted credentials, non-root container execution, and seccomp profiles enabled by default. * **Selective Automation**--Ability to include or exclude specific workload types (e.g., Deployment, StatefulSet) from automation. * **Offline Resilience**--Mutations are cached locally during connectivity issues and synchronized when connection is restored.