Autoscaling Amazon SageMaker AI Endpoints
AWS Certified Machine Learning Engineer – Associate · ML Solution Monitoring, Maintenance, and Security · By Super Admin
Amazon Application Auto Scaling lets SageMaker endpoints scale instance counts based on metrics like invocations or CPU utilization, with cooldowns, step and scheduled scaling, and special considerations for multi-model and multi-container endpoints.
Deployed ML models must handle varying levels of traffic efficiently, and Amazon SageMaker AI endpoints achieve this through horizontal scaling (scale-in, scale-out) of the instances serving inference requests. Amazon Application Auto Scaling automatically adjusts the number of instance replicas for an endpoint based on predefined metrics such as CPU utilization or request count, helping maintain optimal performance while keeping costs under control.
Target Tracking Autoscaling and Cooldowns
Configuring autoscaling requires two components: a scaling target, which sets the boundaries for scaling (for example, a minimum of 1 instance and a maximum of 10), and a scaling policy, which specifies the metric to monitor (such as SageMakerVariantInvocationsPerInstance or CPUUtilization), the target value for that metric, and the conditions for scaling in or out. In a Target Tracking policy built on SageMakerVariantInvocationsPerInstance, a target value of 70 means that if the average invocations per instance rises above 70, instances are added (scale out), and if it drops below 70, instances are removed (scale in). A similar policy can target average CPU utilization (for example, 50%), scaling out when usage exceeds the target and scaling in when it falls below. The ScaleInCooldown and ScaleOutCooldown parameters define the cooldown period, in seconds, after a scaling action, during which no further scaling action of the same type can occur; this prevents rapid, excessive scaling from short-term traffic fluctuations and gives the system time to stabilize.
Other Autoscaling Types
Beyond Target Tracking, two other autoscaling types are relevant for the exam. Step scaling adjusts the number of instances based on predefined steps you define in response to a target threshold triggering event, such as the lower bound, the upper bound, and the amount by which to scale. Scheduled scaling adjusts the number of instances based on a set schedule.
Autoscaling with Multi-Model and Multi-Container Endpoints
Amazon Application Auto Scaling is fully supported by multi-model endpoints (MMEs) and multi-container endpoints (MCEs). When configuring automatic scaling for an MCE using the InvocationsPerInstance metric, it is best practice to ensure the model in each container exhibits similar CPU utilization and latency per inference request. Otherwise, if traffic shifts from a low-CPU-utilization model to a high-CPU-utilization model while the overall invocation volume stays the same, the endpoint will not scale out, resulting in an insufficient number of instances to handle requests to the high-CPU model.
Endpoint Variants and Autoscaling Scope
Autoscaling policies reference a ResourceId such as endpoint/your-endpoint-name/variant/AllTraffic. A variant refers to a different version or configuration of a model deployed to the same endpoint, allowing multiple variants with different instance types, counts, or model versions. Using the suffix variant/AllTraffic applies the policy to all traffic on the endpoint, while specifying a named variant (such as variant/VersionA) allows independent management and scaling of individual model versions—useful for Blue/Green deployments, shadow testing, or canary testing.
Key takeaways
- Target Tracking autoscaling scales endpoints based on metrics like SageMakerVariantInvocationsPerInstance or CPUUtilization compared against a target value.
- ScaleInCooldown and ScaleOutCooldown prevent rapid repeated scaling actions after a scaling event.
- Step scaling reacts to predefined thresholds, while scheduled scaling adjusts capacity on a set schedule.
- For MCEs, models in different containers should have similar CPU utilization and latency to avoid under-scaling when traffic shifts between models.
Discussion (0)
No discussion on this article yet — be the first to comment.
Please sign in to join the discussion.