Kubernetes cost and rightsizing

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/k8s-cost"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/1cc24ac5-8b45-429c-b5c1-f510d7532813/social-card.svg?v=3e7ae33eec8b" alt="Kubernetes cost and rightsizing" width="600" height="315"></a>

Watchtower attributes cluster cost to namespaces and workloads and recommends right-sized requests. It is advisory: Watchtower never changes cluster objects.

Data

The janitor job k8s_cost_rollup runs hourly. For each completed hour it reads the agent metrics k8s.pod.cpu.request, k8s.pod.cpu.usage, k8s.pod.memory.request, k8s.pod.memory.usage, k8s.node.cpu.allocatable and k8s.node.memory.allocatable and writes k8s_cost_1h per cluster, namespace and workload. A workload is the owning Deployment, StatefulSet or DaemonSet (Kind/name); pods without an owner are Pod/name. Pod labels (k8s.pod.label.<key>) are kept for group_by=label:<key>. Memory is in GiB. Rows are kept 400 days.

Price and cost

Set the unit prices once per project:

  • PUT /v1/projects/{prj}/k8s-cost/price-card {"cpu_core_hour": 0.04, "mem_gb_hour": 0.005} (also GET, DELETE; needs ManageProject to write).

Cost of a workload hour = max(request, usage) core-hours times the CPU price plus max(request, usage) GiB-hours times the memory price. A workload is charged for what the node had to reserve for it, or what it really used if that was more. Node capacity no workload claims (allocatable minus the sum above) is reported per cluster as __idle__ ("Idle capacity"). Without a price card every cost is 0; usage is still recorded.

GET /v1/projects/{prj}/k8s-cost?group_by=namespace|workload|label:<k>&from&to (default: the last 7 days, by namespace).

Rightsizing

GET /v1/projects/{prj}/k8s-rightsizing?profile=cost_optimized|balanced|performance.

  • Basis: the last 8 days of hourly rows. CPU is the p95 of hourly usage, memory the maximum.

  • Headroom over the basis: cost_optimized 15%, balanced 30% (default), performance 50%.

  • Floors: 10 m CPU, 32 MiB memory.

  • confidence: high with 7 or more days of data, medium with 3 to 7, low with fewer than 3 days or when the CPU request moved by more than 20% in the window (replicas not steady).

  • hpa_candidate: the mean CPU usage of the busiest hour of the day is at least 3 times that of the quietest.

  • saving_per_month is (current - recommended) x price x 730 h; negative means the workload is under-provisioned. It is 0 when there is no price card (priced: false).

  • yaml_patch is a strategic-merge patch for kubectl patch. Fill in the container name.

Hourly averages understate short bursts, so the p95 is of hourly means. Treat low confidence results as hints only.

Updated