Chip Huyen’s recent presentation at P99 revisits the economics of large‑language‑model (LLM) inference and offers a refreshed set of metrics and levers that can lower spend without buying additional GPUs. Engineers who own the inference stack need to understand these levers because the bulk of a model’s total compute cost is incurred at run‑time, not during the one‑off training phase.
LLM inference optimization: metrics that matter
Huyen emphasizes three latency signals that map directly to cost and user experience:
- Time to first token (TTFT) – the delay before any output reaches the client.
- Time per output token (TPOT) – the average interval between successive tokens, also called inter‑token latency.
- End‑to‑end latency – TTFT plus TPOT multiplied by the number of generated tokens minus one.
For reasoning models, the first generated token may not be visible to the user, so some teams also track “time to publish” – the moment the first token appears on screen. Selecting the primary metric depends on the service‑level objective (SLO) that matters most to your users.
Throughput (requests per window) and goodput (requests that meet latency targets) should be measured together. Huyen’s example targets 200 ms TTFT and 100 ms TPOT, processes ten requests per minute, but only three satisfy both thresholds, illustrating the gap between raw capacity and usable performance.
Model‑level optimization pathways
When you control the model weights – either by hosting the model yourself, fine‑tuning, or training from scratch – two techniques are highlighted:
- Quantization: Reducing weight and activation precision (e.g., from 32‑bit to 8‑bit) shrinks memory footprints and cuts the number of arithmetic operations per inference step. The trade‑off is a modest quality impact, but the approach is broadly applicable across architectures and sizes. Huyen notes that most production deployments have already moved away from full‑precision inference.
- Distillation: Using a larger “teacher” model to generate synthetic training data for a smaller “student” model. The student can approach the teacher’s performance with a fraction of the compute cost. Licensing constraints must be checked, as some providers prohibit using their models to train competing systems.
Service‑level optimization pathways
If the model is treated as a black box, the focus shifts to request handling:
- Batching: Grouping multiple inference calls into a single forward pass. Two strategies are described:
- Static batching – wait until a batch fills before dispatching, maximizing GPU utilization but potentially increasing TTFT for early requests.
- Dynamic batching – dispatch on a timer, offering a compromise between latency and utilization.
- Routing and reuse: Scheduling requests to favor models that are already loaded, and caching intermediate results when possible. These tactics do not alter model weights but can improve goodput without extra hardware.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Practitioners should start by instrumenting the three latency metrics and calculating goodput against existing SLOs. If the model is under your control, evaluate an 8‑bit quantization path first, as it offers the highest ROI with minimal quality loss. Where quantization is not feasible, explore distillation only after confirming licensing allowances.
For black‑box deployments, implement a dynamic batching layer and monitor the impact on TTFT and TPOT. Adjust batch size and timer thresholds to balance utilization against user‑perceived latency. Finally, track the ratio of total throughput to goodput; a large gap signals that further service‑level tuning or model‑level changes are needed.
By aligning measurement, model tweaks, and request‑level orchestration, teams can reduce the dominant inference spend without expanding their hardware footprint.