When benchmarking an Large Language Model (LLM) serving setup, the metric almost everyone reaches for first is throughput: how many requests per second a system can push through. It is easy to measure and aligns with financial metrics regarding dollars spent on infrastructure versus revenue generated from API calls. Consequently, engineers chase this number by tuning batching strategies, raising concurrency limits, and squeezing GPU utilization until request counts climb.
The trouble arises because throughput alone creates a misleading picture of system health. You can observe the metric rising while your service quietly degrades in quality for end-users. I encountered exactly that scenario while fine-tuning vLLM on a single consumer-grade GPU, and the gap was wide enough to warrant deep analysis.
Throughput Counts Plates
To understand why this distinction matters, picture an LLM endpoint as a busy restaurant kitchen. Throughput is simply the number of plates that leave the pass every hour. It tells you how hard your GPU cluster works and whether it has enough capacity to handle load spikes during peak hours.
- High throughput indicates maximum hardware utilization
- Batching strategies increase this metric significantly
This number is vital for architecture decisions, particularly when planning infrastructure costs. However, a kitchen can double its plates per hour and still send out cold food to the wrong tables.
Goodput Counts Happy Diners
The definition of goodput is refreshingly plain: it measures completed requests that also meet your latency targets for every single user. For LLM serving, these specific performance metrics are non-negotiable:
- **Time to First Token (TTFT):** The wait time before the first character appears on screen.
- **Tokens Per Output Second:** How quickly subsequent text streams out once generation begins.
A request that finishes processing but takes ten seconds is useless if your application requires real-time chat responses. A system might achieve 10,000 requests per second throughput by queuing every job and delaying them until the GPU becomes free again to maximize batch efficiency. While this boosts raw numbers, it violates user expectations for interactive applications.
Latency Targets Define Success
In production environments like Kubernetes clusters or managed services on AWS, you must define strict Service Level Objectives (SLOs). If a model takes too long to generate the first token, users abandon sessions before seeing any value. This is why tuning vLLM requires balancing batch size against latency constraints.
Configuration details matter here. You might need to adjust `max_num_batched_tokens` or enable continuous batching in your inference engine settings. These architectural tweaks ensure that new requests are interleaved with ongoing generations rather than waiting for a full queue, directly improving goodput without sacrificing hardware efficiency.
Certification Relevance
Understanding these trade-offs is essential for professionals preparing for advanced cloud certifications such as the AWS Certified Machine Learning – Specialty or Kubernetes certification exams. These credentials often test your ability to architect systems that balance cost, performance, and reliability rather than just raw capacity.
What This Means For You
If you are building an LLM application today, stop optimizing solely for requests per second. Instead of chasing the highest possible throughput number on a dashboard, measure your actual user experience by tracking goodput metrics in real-time monitoring tools like Prometheus or Datadog.