Exa rebuilt the inference stage of its web‑scale vector index to run on AWS Neuron hardware (Inferentia2 and Trainium2) and achieved about a 1.5× boost in embedding throughput. The change matters because higher throughput directly lowers the cost of embedding billions of documents and reduces latency for downstream search and grounding workloads, which are core responsibilities for AI engineers, platform teams, and SREs.
Why the Change Matters
Embedding is the most expensive part of a semantic search pipeline when the corpus reaches tens or hundreds of billions of items. A per‑document cost reduction compounds across the entire dataset, affecting budget, scaling decisions, and service‑level objectives. Faster inference also shortens the window between data discovery and availability in the vector store, improving freshness for real‑time search and agentic applications.
Architecture and Implementation Details
Exa’s pipeline runs asynchronously: crawlers feed raw pages into parse and moderation stages before reaching the embed stage, which writes vectors to an in‑house database. During a backfill of 30 billion priority documents, the embedding load temporarily doubled, prompting a hardware review.
The team selected the smallest Inferentia2 instance type (inf2.xlarge) as a baseline. Each instance provides a single chip with two NeuronCores, but the workload occupies only one core. To avoid idle resources, Kubernetes pods are packed onto larger instances, keeping core utilization uniform across the fleet. The same approach is mirrored on Trainium2, with data‑parallel scaling across NeuronCores.
Initial deployment used a straightforward data‑parallel FP16 model. Subsequent optimizations, applied via the Neuron SDK and in collaboration with the AWS Trainium team, included:
- Weight layout alignment: reorganizing model weights to match the hardware scheduler, eliminating unnecessary transpose operations.
- Autocast down‑casting: allowing the compiler to reduce matrix‑multiply precision while preserving dynamic range, which met Exa’s correctness threshold of 0.999 cosine similarity against the reference model.
Each step contributed incremental gains, culminating in an overall throughput increase of roughly 1.5× over the naive compile.
Operational and Security Considerations
From an operations perspective, the shift to Neuron‑optimized containers means that scaling decisions now hinge on core‑level packing rather than instance count alone. Monitoring must track per‑core utilization to ensure the intended homogeneity and to detect any drift caused by model changes or workload spikes.
Security implications are indirect but worth noting. Running inference on dedicated Neuron hardware isolates the model execution environment from general‑purpose CPUs, potentially reducing the attack surface for side‑channel exploits. However, the pipeline still processes untrusted web content, so existing ingestion sanitization and moderation steps remain the primary defense against malicious payloads.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Practitioners should evaluate whether their embedding workloads can benefit from Neuron‑specific optimizations. Key actions include:
- Profile current throughput on baseline instances to establish a reference point.
- Experiment with weight layout adjustments and enable autocast in the Neuron SDK, verifying that model accuracy stays within acceptable bounds.
- Adopt a core‑packing strategy in orchestration layers to maximize hardware utilization and simplify scaling.
- Review security controls around upstream data ingestion, as hardware changes do not replace the need for robust content validation.
By following this path, teams can achieve measurable cost savings and latency improvements when scaling vector embeddings to billions of documents.


