China Merchants Bank migrated 99% of its AI accelerator inventory onto a single Kubernetes control plane that combines Kueue, KEDA, Prometheus, HAMi and Fluid. The shift raised average accelerator utilization from roughly 35% to over 60% and reduced the cost of processing one million tokens by more than 60%, demonstrating a concrete benefit of composable, cloud‑native scheduling for mixed AI workloads.
Architecture Overview
The bank’s platform treats training, fine‑tuning and online inference as separate workload classes that share a pool of nearly 10,000 heterogeneous accelerator cards. The key pieces are:
- Kueue – handles admission, queuing and quota enforcement for training jobs, preventing premature reservation of capacity.
- KEDA together with Prometheus – provides demand‑driven autoscaling for inference services based on live metrics.
- HAMi – allocates accelerator slices in fine‑grained units across both training and inference paths, enabling the same physical cards to be multiplexed.
- Fluid – accelerates data and model access so that accelerators spend less time waiting on I/O.
On top of this stack, the in‑house Twinkle framework adds a multi‑tenant fine‑tuning layer: five LoRA tenants can share a single base‑model instance, cutting the number of required model replicas from five to one.
Operational Impact
By consolidating 99% of accelerator resources under the unified plane, the bank observed a jump in average utilization to >60%, a figure that directly translates into lower per‑token processing cost. The Twinkle approach further reduced accelerator consumption for LoRA fine‑tuning by 80% while delivering a five‑fold increase in training density. These gains were achieved without changing the underlying hardware, solely by re‑architecting the scheduling and scaling logic.
Additional operational benefits include:
- End‑to‑end observability via
Prometheus, giving operators a single source of truth for both training queue state and inference load. - Dynamic elasticity for inference workloads, allowing rapid scaling up or down in response to traffic spikes.
- Fine‑grained accelerator sharing that reduces idle time and improves overall pool efficiency.
Implications for Platform and Security Operations
The unified control plane centralizes control of heterogeneous accelerators, which simplifies capacity planning but also concentrates responsibility for access control and auditability. Operators should consider:
- Ensuring that admission policies in
Kueueare aligned with organizational governance to prevent unauthorized training jobs from monopolizing resources. - Leveraging the metrics exposed by
Prometheusfor both performance monitoring and security‑related anomaly detection (e.g., unexpected spikes in accelerator usage). - Validating that
HAMislice allocations respect any isolation requirements between teams or workloads, especially when multi‑tenant fine‑tuning is involved. - Reviewing data‑path configurations in
Fluidto confirm that model and dataset caches are not inadvertently exposed across tenant boundaries.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Teams managing large, heterogeneous accelerator fleets should evaluate whether a similar composable stack can replace siloed training and inference pipelines. Start by mapping existing workloads to the four functional blocks (admission, autoscaling, slice allocation, data acceleration) and prototype a small‑scale deployment of Kueue and KEDA on a shared cluster. Monitor utilization and cost metrics through Prometheus to quantify benefits before expanding to the full pool. Finally, incorporate policy checks and observability hooks early to maintain security posture as resources become more tightly coupled.