[网络领域顶会追踪系列#3][NSDI'22] 野生的ML即服务：在大规模异构GPU集群中的负载分析和调度

本系列为个人论文阅读笔记，大多数应该是对顶会摘要的个人菜鸟翻译，旨在让感兴趣的朋友能快速浏览、获取信息。如有错误或心得，欢迎在评论区交流指正！

MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters

背景

随着机器学习技术的持续进步和可用的大规模数据集，科技公司正在部署ML-as-a-Service(MLaaS)云，该云有着许多异构的CPU，来提供一个主机的ML应用。

挑战

然而，在异构GPU集群中运行不同的ML负载会带来许多挑战。

工作

从阿里巴巴6000个GPU的集群中分析了两个月的trace中的特征，解释了集群调度中的挑战，包括低CPU利用率、长队列延迟、难以调度任务、负载不均衡、CPU瓶颈。

描述了作者的解决方案并指出一些待解决的问题。

发布了trace，是在负载和集群规模方面最全面的trace。

原文摘要

With the sustained technological advances in machine learning (ML) and the availability of massive datasets recently, tech companies are deploying large ML-as-a-Service (MLaaS) clouds, often with heterogeneous GPUs, to provision a host of ML applications. However, running diverse ML workloads in heterogeneous GPU clusters raises a number of challenges. In this paper, we present a characterization study of a two-month workload trace collected from a production MLaaS cluster with over 6,000 GPUs in Alibaba. We explain the challenges posed to cluster scheduling, including the low GPU utilization, the long queueing delays, the presence of hard-to-schedule tasks demanding high-end GPUs with picky scheduling requirements, the imbalance load across heterogeneous machines, and the potential bottleneck on CPUs. We describe our current solutions and call for further investigations into the challenges that remain open to address. We have released the trace for public access, which is the most comprehensive in terms of the workloads and cluster scale.