As the demand for generative AI continues to grow, developers and enterprises seek more flexible, cost-effective, and powerful accelerators to meet their needs. Today, we are thrilled to announce the availability of G7e instances powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on Amazon SageMaker AI. You can provision nodes with 1, 2, 4, and 8 RTX PRO 6000 GPU instances, with each GPU providing 96 GB of GDDR7 memory. This launch provides the capability to use a single-node GPU, G7e.2xlarge instance to host powerful open source foundation models (FMs) like GPT-OSS-120B, Nemotron-3-Super-120B-A12B (NVFP4 variant), and Qwen3.5-35B-A3B, offering organizations a cost-effective and high-performing option. This makes it well suited for those looking to improve costs while maintaining high performance for inference workloads.
Hardware Specifications and Memory Architecture
The key highlights for G7e instances include twice the GPU memory compared to G6e instances, enabling deployment of large language models (LLMs) in FP16 up to: 35B parameter model on a single GPU node (G7e.2xlarge), 150B parameter model on a 4 GPU node (G7e.24xlarge), and 300B parameter model on an 8 GPU node (G7e.48xlarge). Up to 1600 Gbps of networking throughput is available, and up to 768 GB GPU Memory is available on G7e.48xlarge. Amazon Elastic Compute Cloud (Amazon EC2) G7e instances represent a significant leap in GPU-accelerated inference on the cloud. They deliver up to 2.3x inference performance compared to the previous-generation G6e instances. Each G7e GPU provides 1,597 Gbps of memory bandwidth, which is critical for reducing latency in high-throughput inference scenarios.
Model Deployment Strategies for Large Language Models
For cloud engineers preparing for AWS certifications such as the AWS ML Specialty or SAA-C03, understanding the memory requirements for different model sizes is essential. The G7e instances allow for the deployment of massive language models with significantly reduced costs. For example, a single G7e.2xlarge instance can host a 35B parameter model, while a G7e.48xlarge instance can host a 300B parameter model. This scalability is crucial for organizations looking to optimize their infrastructure for inference workloads. The architecture supports various quantization formats, including NVFP4, which helps in maintaining performance while reducing memory footprint. This flexibility is particularly useful when dealing with models like Nemotron-3-Super-120B-A12B, where memory efficiency is paramount.
Networking and Throughput Considerations
Up to 1600 Gbps of networking throughput is a significant improvement over previous generations, enabling faster data transfer between nodes in a distributed inference setup. This high bandwidth is essential for multi-node deployments where models are split across multiple GPUs. For DevOps professionals managing Kubernetes clusters on AWS, this networking capability ensures that inference requests are handled efficiently without bottlenecks. The G7e instances are designed to handle the high data rates required for real-time generative AI applications, making them suitable for production environments where latency is a critical factor.
What This Means For You
For professionals studying for AWS certifications, the G7e instances offer a practical example of how hardware advancements can impact model deployment strategies. The ability to deploy larger models on fewer nodes can lead to significant cost savings and improved performance. For those preparing for the AWS ML Specialty certification, understanding the trade-offs between model size, memory, and throughput is crucial. The G7e instances provide a robust platform for experimenting with and deploying state-of-the-art generative AI models. By leveraging these instances, organizations can stay ahead in the competitive landscape of generative AI, ensuring they have the necessary infrastructure to support their AI initiatives effectively.