The AWS CDK now offers a reusable construct that swaps the built‑in AWS::ApplicationAutoScaling::ScalableTarget and AWS::ApplicationAutoScaling::ScalingPolicy resources for three custom resources. These resources automatically deregister the scalable target, apply the endpoint update, then re‑register the target and attach the scaling policy, all within one CloudFormation stack operation. Engineers who need to change instance types, counts, or environment variables on a SageMaker endpoint can now do so without manual CLI work or risk of a broken stack.
Why Native Auto Scaling Breaks on Endpoint Updates
When a SageMaker endpoint is created with native auto‑scaling resources, changing the endpoint configuration triggers one of three failure patterns:
- Updating the endpoint while the scalable target remains registered causes SageMaker to reject the request with an error stating the variant must be deregistered first.
- Manually deregistering the target via CLI and then updating only the endpoint configuration lets the deployment succeed, but CloudFormation does not re‑register the target because the template’s scaling parameters appear unchanged.
- Manually deregistering and then changing scaling parameters leads CloudFormation to attempt an update on a non‑existent
AWS::ApplicationAutoScaling::ScalableTarget, resulting in a stack that fails to roll back and requires manual recovery.
CDK Construct Workflow
The construct enforces a deterministic order using node.add_dependency calls. The sequence is:
- Deregister – a custom resource that calls the SageMaker API to remove the scalable target. On first deployment the target is absent, so the handler treats the error as a no‑op.
- Endpoint update – the standard CDK‑managed
AWS::SageMaker::Endpointresource applies the new configuration. - Register Scalable Target – another custom resource waits until the endpoint reaches
InServicebefore creating the scalable target. - Attach Scaling Policy – a final custom resource waits for the target to become queryable, then applies the target‑tracking policy.
Implementation Details
Each custom resource uses the CDK provider pattern with separate on_event and is_complete handlers. The on_event handler returns immediately; the is_complete handler is polled every 30 seconds until the required state is observed. Timeouts are bounded (1 hour for registration, 15 minutes for policy attachment) to avoid indefinite Lambda execution.
A ConfigHash property is added to force CloudFormation to re‑invoke the Lambda functions whenever any configuration value changes. This ensures that updates to instance type, count, or scaling thresholds trigger the full orchestration cycle.
Operational and Security Considerations
Deploying the construct introduces Lambda functions that must have permission to call SageMaker and Application Auto Scaling APIs. Teams should audit the IAM role to follow least‑privilege principles.
Because the deregister step runs before the endpoint update, any failure in the deregistration handler could block the entire stack. Monitoring the custom resource logs and setting up CloudWatch alarms for failed invocations is advisable.
The explicit dependency chain prevents CloudFormation from attempting to update a missing scalable target, reducing the chance of UPDATE_ROLLBACK_FAILED states. However, if a rollback occurs after the registration step, the custom resource will attempt to clean up the target again, so idempotent error handling in the Lambda code is essential.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopting the CDK construct eliminates manual CLI steps and protects stacks from inconsistent auto‑scaling state after endpoint changes. Engineers should integrate the construct into their CI/CD pipelines, verify IAM permissions for the supporting Lambdas, and add observability around the custom resource lifecycle. Doing so yields reliable, single‑transaction updates to SageMaker endpoints while preserving auto‑scaling behavior.
