Salesforce Achieves Cost Reduction and Resilience with SageMaker AI Insights
Discover how Salesforce tackled the challenges of AI model availability by implementing a Multi-AZ strategy with Amazon SageMaker, ensuring compliance and resilience while significantly reducing infrastructure costs.
Key Facts
- Salesforce achieved 8x infrastructure cost reduction using SageMaker ICs, enhancing financial efficiency.
- New SchedulingConfig parameter mitigates single points of failure, improving operational resilience.
- Multi-AZ compliance ensures no model downtime from AZ failures, strengthening competitive positioning.
- Pre-provisioning GPU capacity is crucial for balanced deployments, revealing a vulnerability in on-demand reliance.
- Continuous monitoring with SageMaker AI Insights is essential for maintaining HA, indicating strategic oversight needs.
Summary
Summary
Salesforce aimed to enhance the availability of Agentforce, their AI foundation for agents, across multiple Availability Zones (AZs). The challenge was to eliminate single points of failure while maintaining cost efficiency. By implementing Amazon SageMaker AI Inference Components with the new SchedulingConfig parameter, Salesforce achieved Multi-AZ high availability, resulting in an 8x reduction in infrastructure costs.
Background
Salesforce, a leading customer relationship management (CRM) platform, operates in the technology industry and serves millions of users globally. Before the deployment of SageMaker AI Inference Components, Salesforce faced challenges in ensuring high availability for their AI models, particularly in meeting compliance requirements for multi-AZ support.
Challenge
The primary issue was the risk of single points of failure in the deployment of inference components. The default placement algorithm of SageMaker did not guarantee that model copies were evenly distributed across AZs, which posed compliance risks and potential outages during instance or AZ failures.
Solution
Salesforce utilized the SchedulingConfig parameter in the CreateInferenceComponent API to control the placement of inference component copies across instances and AZs. Key features included the AvailabilityZoneBalance for even distribution and the SPREAD placement strategy to maximize fault isolation. This allowed Salesforce to deploy models with multiple copies across different AZs, ensuring compliance with their 2-AZ support mandate.
Results
Salesforce achieved several key outcomes through this deployment:
- Multi-AZ high availability compliance for all model deployments.
- Elimination of single points of failure, ensuring that no model could be fully taken offline due to an instance or AZ failure.
- Continued infrastructure cost savings with an 8x reduction in costs through multi-model co-hosting.
- Resilient scaling operations that preserved multi-AZ distribution during scale-up and scale-down activities.
- Persistent high availability during model updates, preventing disruptions in service.
Key Insights
Enterprise AI teams should prioritize high availability at the inference component level, not just at the endpoint level. Utilizing the SchedulingConfig with SPREAD and AvailabilityZoneBalance is recommended for high availability workloads. Pre-provisioning capacity with On-Demand Capacity Reservations (ODCR) is crucial for achieving balanced AZ placement, and continuous monitoring of IC distribution is essential to maintain reliability.
Customer Testimonial
No direct quote is available from the source material.
Entities Mentioned
Companies
Products
Technologies
Key Concepts
Definitions
- Multi-AZ HA
- Multi-AZ High Availability refers to the deployment of applications across multiple Availability Zones to ensure resilience and uptime.
- SageMaker
- Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning models quickly.
- IC Placement
- IC Placement is a feature that allows users to control the distribution of inference component copies across instances and Availability Zones.
- SPREAD
- SPREAD is a placement strategy that distributes copies of a model across as many instances as possible for maximum fault isolation.
- MaxImbalance
- MaxImbalance is a parameter that defines the maximum allowable difference in the number of copies of a model between any two Availability Zones.
Use Cases
- →Deploying AI models with Multi-AZ HA compliance
- →Cost reduction through GPU co-hosting
- →Maintaining availability during scaling operations
- →Monitoring IC distribution for compliance
- →Implementing fault isolation strategies
- →Using SchedulingConfig for model deployment
Frequently Asked Questions
What is Multi-AZ HA and why is it important?
Multi-AZ High Availability ensures that applications remain operational even if one Availability Zone fails. This is crucial for maintaining uptime and meeting compliance requirements.
How does Salesforce achieve cost efficiency with SageMaker?
Salesforce reduces infrastructure costs by co-hosting multiple models on shared GPUs, achieving an 8x reduction in costs while maintaining high availability.
What is the role of the SchedulingConfig parameter?
The SchedulingConfig parameter allows users to control the placement of inference components across instances and Availability Zones, ensuring compliance with high availability requirements.
Why should CopyCount never be set to 1 for HA-critical models?
Setting CopyCount to 1 means that the model can only exist in one Availability Zone, which violates the 2-AZ compliance requirement and creates a single point of failure.
What monitoring tools does SageMaker provide for IC placement?
SageMaker AI Insights offers metrics such as AZ skew and IC copy count per AZ, which help users monitor and maintain Multi-AZ high availability and detect any imbalances.