AssemblyAI has replaced an all‑On‑Demand GPU fleet with a blended pool of EC2 Spot and On‑Demand instances, orchestrated through Amazon ECS capacity providers. The change lets engineers meet latency SLAs at a fraction of the previous compute cost.
Architecture Overview
The solution runs an Amazon ECS cluster inside a VPC that spans multiple Availability Zones. Each ECS task runs on a GPU‑backed EC2 instance, pulls transcription jobs from an Amazon SQS queue, reads the source audio from Amazon S3, and writes the completed transcript back to a results store. When a Spot instance receives a two‑minute interruption notice, Amazon EventBridge invokes a Lambda function (the Spot Interruption Handler) that re‑queues the in‑flight job to the main SQS queue. Jobs that exceed retry limits are routed to a dead‑letter queue for manual investigation. Amazon CloudWatch collects metrics on fleet health, Spot interruptions, and queue depth, providing the observability needed to operate at scale.
ECS Capacity Providers and Spot Diversification
Two capacity providers are defined per cluster: an always‑on On‑Demand provider that guarantees a baseline of GPU capacity, and a Spot provider that spans several GPU instance families—g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge, and g6.2xlarge. Each provider is linked to an Auto Scaling group. ECS managed scaling adjusts the number of instances based on the weighted capacity‑provider strategy and the number of pending tasks. By diversifying across instance families and Availability Zones, the architecture maximises the chance of acquiring Spot capacity during demand spikes.
Spot Interruption Handling
When EC2 emits an interruption notice, EventBridge triggers the Lambda handler, which extracts the job identifier from the instance metadata and pushes the job back onto the primary SQS queue. The retry loop allows another healthy task to pick up the work without violating the service‑level agreement. The dead‑letter queue isolates jobs that repeatedly fail, enabling engineers to investigate root causes without affecting the main processing flow.
Operational and Security Implications
Operating a Spot‑heavy fleet requires continuous monitoring of interruption rates and queue latency; CloudWatch alarms should be tuned to detect rising re‑queue volumes. Because customers in regulated sectors must keep audio data within specific regions, the deployment is duplicated in both US West (us‑west‑2) and Europe (eu‑west‑2). This regional split enforces data residency while still leveraging the same Spot‑centric pattern. Network isolation is provided by the VPC, and the S3 buckets used for input and output remain confined to the same region, reducing cross‑region data movement.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopt an ECS capacity‑provider strategy that mixes On‑Demand and Spot capacity, and include multiple GPU instance families to improve Spot acquisition rates. Implement a queue‑based retry loop with a Lambda interruption handler to recover jobs transparently. Deploy the stack in the regions required for data residency and instrument CloudWatch metrics for Spot interruptions and queue health. Following this pattern can deliver substantial cost savings without compromising the latency guarantees of GPU‑intensive inference workloads.
