[网络领域顶会追踪系列#17][NSDI'22]Check-N-Run: 一个针对训练深度学习推荐模型的checkpointing系统
本系列为个人论文阅读笔记，大多数应该是对顶会摘要的个人菜鸟翻译，旨在让感兴趣的朋友能快速浏览、获取信息。如有错误或心得，欢迎在评论区交流指正！
Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
背景
checkpoints在训练长时间运行的机器学习（ML）模型中起到很重要的作用。checkpoins为ML模型拍一个快照，并且存储它在一个不易失的内存中，这样checkpoints就可以用于故障恢复，以保证快速的训练过程。此外，checkpoints还被用于在线训练，以通过持续学习提高推理预测的准确性。
挑战
鉴于模型尺寸庞大且不断增加，checkpoint的频率被存储的写带宽和容量所限制。当checkpoint被远端存储所维护（这也是一些工业化配置的场景），checkpoints频率也会被网络带宽所限制。
工作
作者提出了Check-N-Run，一个facebook的训练大型ML模型的可扩展checkpointing系统。
因为Check-N-Run应用于长时间运行的ML任务，作者聚焦于checkpointing推荐模型，它是有着TB级模型大小的当前最大的ML模型。
Check-N-Run主要使用了两个技术来解决尺寸和带宽挑战：
- 首先，它使用差异化checkpointing，可以追踪和检查模型的修改部分。差异化checkpointing在推荐模型的情景中是十分重要的，因为只有一部分模型（以嵌入表存储）会在每次迭代中被更新。
- 其次，Check-N-Run在不降低训练准确度的前提下，利用了量化技术来极大地减小了checkpoint的大小。
实验
这些技术让Check-N-Run在facebook的真实模型上，将需要的带宽减小了6到17倍、需要的容量减小了2.5到8倍，因此极大地提升了checkpoint的能力，并且减小了所有权（ownership）的总开销。
原文摘要
Checkpoints play an important role in training long running machine learning (ML) models. Checkpoints take a snapshot of an ML model and store it in a non-volatile memory so that they can be used to recover from failures to ensure rapid training progress. In addition, they are used for online training to improve inference prediction accuracy with continuous learning. Given the large and ever-increasing model sizes, checkpoint frequency is often bottlenecked by the storage write bandwidth and capacity. When checkpoints are maintained on remote storage, as is the case with many industrial settings, they are also bottlenecked by network bandwidth. We present Check-N-Run, a scalable checkpointing system for training large ML models at Facebook. While Check-N-Run is applicable to long running ML jobs, we focus on checkpointing recommendation models which are currently the largest ML models with Terabytes of model size. Check-N-Run uses two primary techniques to address the size and bandwidth challenges. First, it applies differential checkpointing, which tracks and checkpoints the modified part of the model. Differential checkpointing is particularly valuable in the context of recommendation models where only a fraction of the model (stored as embedding tables) is updated on each iteration. Second, Check-N-Run leverages quantization techniques to significantly reduce the checkpoint size, without degrading training accuracy. These techniques allow Check-N-Run to reduce the required write bandwidth by 6-17x and the required capacity by 2.5-8xon real-world models at Facebook, and thereby significantly improve checkpoint capabilities while reducing the total cost of ownership.