[网络领域顶会追踪系列#8][NSDI'22]Cocktail：一种云上模型服务的多维度优化方法
本系列为个人论文阅读笔记，大多数应该是对顶会摘要的个人菜鸟翻译，旨在让感兴趣的朋友能快速浏览、获取信息。如有错误或心得，欢迎在评论区交流指正！
Cocktail: A Multidimensional Optimization for Model Serving in Cloud
背景
随着为广泛的应用服务调整ML模型的需求的日益增长，在公有云环境下，服务这些模型的框架能够在最小的延迟和较小的部署开销的情况下提供高预测准确度变得很重要。
挑战
除了有着高延迟，之前在这个领域的工作被各个模型提供的准确度严重限制住了。然而，实时动态地选择合适的模型以满足准确度是一个很重要的问题。
工作
针对此，提出了CockTail，一个低成本、基于合并计算的模型服务框架。
CockTail由两个核心组件组成：
- 一个动态的模型选择框架，可在满足准确度和延迟要求的同时，减少合并时的模型数
- 一个自适应的资源管理（Resource Management, RM）框架，可部署一个分布式的主动自动缩放测量，来高效地为模型分配资源。RM框架利用瞬时的VM实例来减少在公有云的部署开销。
实验
在AWS EC2平台上实现的原型和基于多种负载的详尽的实验表明，CockTail与最先进的模型服务框架相比，能在提供2x的时延减小和对于最高96%的请求能满足目标准确度要求的同时， 可以减小1.45x的部署开销。
原文摘要
With a growing demand for adopting ML models for a variety of application services, it is vital that the frameworks serving these models are capable of delivering highly accurate predictions with minimal latency along with reduced deployment costs in a public cloud environment. Despite high latency, prior works in this domain are crucially limited by the accuracy offered by individual models. Intuitively, model ensembling can address the accuracy gap by intelligently combining different models in parallel. However, selecting the appropriate models dynamically at runtime to meet the desired accuracy with low latency at minimal deployment cost is a nontrivial problem. Towards this, we propose Cocktail, a cost effective ensembling-based model serving framework. Cocktail comprises of two key components: (i) a dynamic model selection framework, which reduces the number of models in the ensemble, while satisfying the accuracy and latency requirements; (ii) an adaptive resource management (RM) framework that employs a distributed proactive autoscaling policy combined with importance sampling, to efficiently allocate resources for the models. The RM framework leverages transient virtual machine (VM) instances to reduce the deployment cost in a public cloud. A prototype implementation of Cocktail on the AWS EC2 platform and exhaustive evaluations using a variety of workloads demonstrate that {Cocktail} can reduce deployment cost by 1.45x, while providing 2x reduction in latency and satisfying the target accuracy for up to 96% of the requests, when compared to state-of-the-art model-serving frameworks.