Controlling Multi‑AZ Placement for SageMaker Inference Components

Amazon SageMaker now lets you dictate how Inference Component (IC) copies are spread across Availability Zones using the new SchedulingConfig parameter in the CreateInferenceComponent API. This control lets teams that need strict multi‑AZ resilience—such as Salesforce’s Agentforce—move beyond the default cost‑centric placement and meet compliance requirements for high availability.

What changed: explicit AZ‑aware placement

Previously, SageMaker’s placement algorithm treated each IC deployment as an isolated operation, distributing copies evenly across instances but without regard for AZ balance. The result could be an uneven distribution that left a model vulnerable to a single‑instance or AZ outage. The addition of SchedulingConfig introduces two sub‑parameters that address this gap:

• AvailabilityZoneBalance – defines how evenly copies must be spread across AZs and lets you set a tolerance (MaxImbalance) for deviation.
• PlacementStrategy – within each AZ you can choose SPREAD (maximally distribute copies across instances) or BINPACK (concentrate copies for higher utilization).

How to configure multi‑AZ HA for ICs

Below is a minimal Python example that creates an IC with four copies, a SPREAD strategy, and permissive AZ balancing that allows at most one copy difference between zones. The endpoint already runs two instances in each of two AZs, so the four copies end up as two per AZ.

response = client.create_inference_component(
    InferenceComponentName='salesforce-llm-ic-ha',
    EndpointName='salesforce-multiaz-endpoint',
    VariantName='AllTraffic',
    Specification={
        'ModelName': 'salesforce-einstein-llm-v2',
        'ComputeResourceRequirements': {
            'NumberOfAcceleratorDevicesRequired': 1,
            'MinMemoryRequiredInMb': 65536
        },
        'DataCacheConfig': {'EnableCaching': True},
        'SchedulingConfig': {
            'PlacementStrategy': 'SPREAD',
            'AvailabilityZoneBalance': {
                'EnforcementMode': 'PERMISSIVE',
                'MaxImbalance': 1
            }
        }
    },
    RuntimeConfig={'CopyCount': 4}
)

For lighter models that only need two copies, setting MaxImbalance to 0 forces a strict 1‑copy‑per‑AZ distribution:

# Example for a 2‑copy model
'SchedulingConfig': {
    'PlacementStrategy': 'SPREAD',
    'AvailabilityZoneBalance': {
        'EnforcementMode': 'PERMISSIVE',
        'MaxImbalance': 0
    }
},
RuntimeConfig={'CopyCount': 2}

Operational impact of scaling with AZ balance

When you scale an IC, SageMaker respects the same SchedulingConfig values. A scale‑out adds new copies to the AZ that currently has fewer copies, preserving balance. A scale‑in removes copies symmetrically, avoiding a situation where one AZ is left with a single copy. The following call doubles the copy count to eight, resulting in four copies per AZ:

update_response = client.update_inference_component(
    InferenceComponentName='salesforce-llm-ic-ha',
    RuntimeConfig={'CopyCount': 8}
)

Note that setting CopyCount to 1 defeats the HA goal, because a single copy can reside in only one AZ and would immediately violate a 2‑AZ compliance rule.

Related CloudNinjas coverage: AWS.

What This Means For Practitioners

With SchedulingConfig you can align SageMaker IC deployments with existing multi‑AZ HA policies without sacrificing the cost benefits of shared‑GPU inference. Engineers should audit existing IC deployments for AZ imbalance, add explicit SchedulingConfig where needed, and incorporate the parameter into any automated provisioning or scaling pipelines. Monitoring should include copy distribution metrics to verify that the actual placement matches the intended balance, especially after manual or automated scale events. Finally, consider pairing the placement configuration with an endpoint ScaleInPolicy that uses the CONSOLIDATION strategy to keep background rebalancing consistent over time.