China Merchants Bank Achieves Major Efficiency Gains in AI Training
China Merchants Bank's groundbreaking approach to unifying AI workloads on Kubernetes not only won the CNCF End User Case Study Contest but also revolutionized its operational efficiency, cutting costs dramatically while optimizing resource utilization.
Key Facts
- China Merchants Bank's AI platform increased accelerator utilization from 35% to over 60%, enhancing efficiency.
- Cost of processing 1 million tokens dropped by over 60%, indicating significant financial savings.
- Unified control plane allows sharing of 10,000 accelerator cards, revealing strong resource optimization.
- Twinkle framework boosts training density fivefold, showcasing competitive advantage in AI resource management.
- Future plans focus on dynamic training concurrency and serverless inference, signaling strategic innovation.
Summary
Summary
China Merchants Bank, a leading commercial bank in China, faced challenges in optimizing its AI training and inference processes. By implementing a unified Kubernetes control plane, the bank improved accelerator compute utilization from 35% to over 60% and reduced the cost of processing 1 million tokens by more than 60%.
Background
China Merchants Bank operates in the financial services industry and is recognized as one of China's leading commercial banks. Before deploying their AI infrastructure solution, the bank struggled with low utilization rates of their accelerator compute resources, which hindered their ability to efficiently manage AI workloads.
Challenge
The bank aimed to address the inefficiencies in AI training and inference, where different demands from distributed training and online inference were not being met effectively. This resulted in underutilized hardware and increased operational costs.
Solution
The bank's AI infrastructure team developed a unified control plane by integrating Kubernetes with Kueue, KEDA, Prometheus, HAMi, and Fluid. This architecture allowed nearly 10,000 heterogeneous accelerator cards to be shared for model training, fine-tuning, and online inference. The solution also included the in-house Twinkle training framework, enabling five LoRA tenants to share a single base model instance, significantly reducing resource usage.
Results
The implementation led to a significant increase in average accelerator compute utilization, rising from 35% to over 60%. Additionally, the cost of processing 1 million tokens was reduced by more than 60%. The Twinkle framework cut accelerator resource usage for multi-tenant setups by 80% while increasing training density fivefold.
Key Insights
Organizations can achieve substantial efficiency gains by unifying their AI training and inference processes through a composable architecture. Leveraging Kubernetes and other cloud-native technologies can optimize resource utilization and reduce operational costs, particularly in environments with diverse workload demands.
Customer Testimonial
"At China Merchants Bank, we unlock the potential of hardware through software innovation," said PeiXiang Tan, AI Infrastructure Architect, China Merchants Bank. "Built on a unified cloud-native foundation, our platform brings together heterogeneous compute pooling, training job scheduling, elastic inference scaling, accelerated access to data and models, and end-to-end observability."
Entities Mentioned
Companies
Products
People
Organizations
Key Concepts
Definitions
- Kubernetes
- An open-source platform designed to automate deploying, scaling, and operating application containers.
- KEDA
- Kubernetes-based Event Driven Autoscaling, a project that allows applications to scale based on events.
- Prometheus
- An open-source monitoring and alerting toolkit widely used with Kubernetes.
- multi-tenant fine-tuning
- A method that allows multiple users to share a single model instance for training, optimizing resource usage.
- cloud-native
- An approach to building and running applications that fully exploit the advantages of the cloud computing delivery model.
Use Cases
- →AI workloads in financial sectors
- →Unified control plane for AI training
- →Dynamic multi-tenant training concurrency
- →Cost-effective processing of tokens
- →Efficient resource allocation for training and inference
- →Accelerated access to datasets and model weights
Frequently Asked Questions
What is the significance of the CNCF End User Case Study Contest?
The CNCF End User Case Study Contest recognizes innovative implementations of cloud-native technologies. Winning this contest highlights the effectiveness of a company's approach to leveraging cloud-native solutions for operational efficiency.
How did China Merchants Bank improve accelerator compute utilization?
By implementing a unified Kubernetes control plane, China Merchants Bank was able to consolidate nearly 10,000 accelerator cards, increasing average utilization from 35% to over 60%. This was achieved through effective resource management and scheduling.
What role does the Twinkle training framework play?
The Twinkle training framework allows multiple tenants to share a single base model instance, significantly reducing resource usage by 80% while enhancing training density. This innovation supports efficient multi-tenant fine-tuning.
What are the future plans for China Merchants Bank's AI infrastructure?
China Merchants Bank plans to enhance its AI infrastructure by dynamically adjusting multi-tenant training concurrency and extending KEDA for serverless inference. They aim to improve capacity management and support for heterogeneous accelerators.
Why is cloud-native infrastructure important for AI workloads?
Cloud-native infrastructure is crucial for AI workloads as it provides scalability, flexibility, and efficient resource utilization. It allows organizations to manage diverse workloads effectively while minimizing costs and maximizing performance.