> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kubex.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Reactive Policies

> Experimental: GPU/KAI fields and related custom resources are subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`GpuReactivePolicy` and `ClusterGpuReactivePolicy` emit GPU reactive recommendations (upsize and downsize) from Prometheus utilization.

## Behavior

* Baseline is the live GPU allocation before first upsize and is persisted per container.
* `spec.scope.containers` limits baseline state, metrics, aggregation, and redistribution to exact names. Empty means all regular containers and restartable init containers. Narrowing scope removes old recommendation and baseline entries for excluded containers.
* By default, only one same-kind GPU reactive policy contributes to a workload. With `globalConfiguration.multiPolicyContainerRightsizingEnabled: true`, separate container/resource targets can come from different policies, and `PolicyEvaluation` resolves overlaps. See [Multi-Policy Container Rightsizing](https://docs.kubex.ai/docs-kubex/Content/Kubex/automation-engine/multi-policy-container-rightsizing.md).
* Pod metrics are considered only after the pod age reaches `spec.minPodMetricsAge` (default `15m`).
* The policy evaluates two required GPU signals: `spec.metrics.compute` and `spec.metrics.memory`.
* Each metric declares its Prometheus value interpretation through `spec.metrics.<signal>.prometheus.interpretation`. `fullGPU` treats the value as a percentage of one whole GPU. `currentAllocation` treats it as a percentage of the container's current GPU allocation.
* Each metric is evaluated from per-pod aggregate GPU usage: the controller sums inferred GPU usage across all GPU containers in the pod, and compares that pod total against the summed current GPU allocation for those same containers.
* Upsize evaluation is pod-level per metric: if any eligible pod total exceeds that metric's threshold, that metric can request an upsize.
* Scale-back evaluation is owner-wide per metric: all eligible pod totals must stay below `currentAllocation * (spec.metrics.<signal>.scaleBack.thresholdPercent/100)` over `spec.metrics.<signal>.scaleBack.metricsWindow`, and every included container in those pods must have samples.
* `spec.metrics.<signal>.upsize.metricsWindow` and `spec.metrics.<signal>.scaleBack.metricsWindow` must be at least `1m`.
* Pods without `status.startTime` are treated as ineligible for metric checks.
* Containers missing a current GPU allocation are excluded from aggregate math and omitted from the emitted recommendation for that reconcile.
* Recommendations are emitted as GPU **requests** (`gpu`) and consumed by normal policy evaluation.
* Upsize target is the hottest eligible pod total plus `spec.metrics.<signal>.upsize.headroomPercent` (default `20`), capped by `spec.metrics.<signal>.upsize.maxPercent` relative to the included containers' summed baseline.
* Scale-back target is the hottest eligible pod total plus `spec.metrics.<signal>.scaleBack.headroomPercent` (default `20`), floored to the included containers' summed baseline.
* The controller compares the compute and memory recommendations and keeps the higher desired total.
* A lower recommendation is accepted when the existing recommendation was driven by the same metric.
* A lower cross-metric recommendation is accepted only if the previously driving metric also emits in that reconcile and its candidate is at or below the accepted lower total. Equal lower totals keep the existing driving metric to avoid ownership churn.
* Existing recommendations that predate driving-metric metadata do not lower until a non-decreasing recommendation establishes an owner metric.
* After a workload total is chosen, it is redistributed back to the selected pod's containers in proportion to their current GPU allocations.
* If neither upsize nor scale-back produces an accepted recommendation and current allocation still differs from baseline, the controller reuses the previous recommendation when present; otherwise it emits nothing.
* Workloads matched by GPU reactive policies are reevaluated periodically using `GlobalConfiguration.spec.gpuReactiveCheckInterval` (default `1m`).

## Namespaced Spec

```yaml theme={null}
apiVersion: rightsizing.kubex.ai/v1alpha1
kind: GpuReactivePolicy
metadata:
  name: gpu-reactive
  namespace: default
spec:
  experimental:
    gpuKaiContract: v1alpha1-2026-07
  scope:
    containers:
      - inference
    labelSelector:
      matchLabels:
        app: my-gpu-app
  minPodMetricsAge: 15m
  metrics:
    compute:
      upsize:
        thresholdPercent: 125
        metricsWindow: 10m
        headroomPercent: 20
        maxPercent: 200
      scaleBack:
        thresholdPercent: 60
        metricsWindow: 10m
        headroomPercent: 20
      prometheus:
        metric: kubex_gpu_container_sm_utilization_percent
        interpretation: fullGPU
        namespaceLabel: namespace
        podLabel: pod
        containerLabel: container
    memory:
      upsize:
        thresholdPercent: 125
        metricsWindow: 10m
        headroomPercent: 20
        maxPercent: 200
      scaleBack:
        thresholdPercent: 60
        metricsWindow: 10m
        headroomPercent: 20
      prometheus:
        metric: kubex_gpu_container_memory_footprint_percent
        interpretation: currentAllocation
        namespaceLabel: namespace
        podLabel: pod
        containerLabel: container
  automationStrategyRef:
    name: sample-automation-strategy
```

## Cluster Spec

```yaml theme={null}
apiVersion: rightsizing.kubex.ai/v1alpha1
kind: ClusterGpuReactivePolicy
metadata:
  name: gpu-reactive-cluster
spec:
  experimental:
    gpuKaiContract: v1alpha1-2026-07
  scope:
    containers:
      - inference
    namespaceSelector:
      operator: In
      values: ["*"]
    labelSelector:
      matchLabels:
        app: my-gpu-app
  minPodMetricsAge: 15m
  metrics:
    compute:
      upsize:
        thresholdPercent: 125
        metricsWindow: 10m
        headroomPercent: 20
        maxPercent: 200
      scaleBack:
        thresholdPercent: 60
        metricsWindow: 10m
        headroomPercent: 20
      prometheus:
        metric: kubex_gpu_container_sm_utilization_percent
        interpretation: fullGPU
        namespaceLabel: namespace
        podLabel: pod
        containerLabel: container
    memory:
      upsize:
        thresholdPercent: 125
        metricsWindow: 10m
        headroomPercent: 20
        maxPercent: 200
      scaleBack:
        thresholdPercent: 60
        metricsWindow: 10m
        headroomPercent: 20
      prometheus:
        metric: kubex_gpu_container_memory_footprint_percent
        interpretation: currentAllocation
        namespaceLabel: namespace
        podLabel: pod
        containerLabel: container
  automationStrategyRef:
    name: sample-cluster-automation-strategy
```

## Global Prometheus Settings

Configure controller-wide Prometheus endpoint/timeouts via `GlobalConfiguration`:

```yaml theme={null}
apiVersion: rightsizing.kubex.ai/v1alpha1
kind: GlobalConfiguration
metadata:
  name: global-config
spec:
  prometheus:
    url: http://prometheus.monitoring.svc:9090
    requestTimeout: 30s
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.