知识卡片

普通读书笔记卡 · 7010.d · 生成模型:DeepSeek-V4.1-Flash

尾延迟与加副本的局限

理解等级:高级

一个常被混淆的关键点:靠增加模型副本数来做并行化,通常并不是预测延迟的解法,因为把单个样本送过模型的端到端时间不会因为「多几份模型可用」而变短——副本解决的是吞吐与容量,不是单次路径。因此延迟过高时正确的方向是更强硬件或让模型本身计算更便宜,而不是无脑扩副本。此外真实系统因网络效应与整体负载会产出延迟分布而非单一数值,只看平均延迟会掩盖一部分请求被丢弃;更有用的是看尾延迟,即最差的那百分之几。这与第 7 章「训练系统吞吐量」形成对照:服务侧要盯分布与尾部,训练侧要盯长期吞吐趋势。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Serving · 查看原书依据

      However, it is important to note that parallelization by creating a larger number of model replicas is usually not a solution to prediction latency, as the end-to-end time it takes to send a single example through the model isn’t affected by simply having more versions of the model available. Real systems often produce a distribution of latency values, due to network effects and overall system load.

    参考来源

    - 位置:《Reliable Machine Learning》第08章《Serving》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch08.xhtml) - 结论依据:原文明确阐述该主题,并给出定义、实践或反例。 - 原始内容:However, it is important to note that parallelization by creating a larger number of model replicas is usually not a solution to prediction latency, as the end-to-end time it takes to send a single example through the model isn’t affected by simply having more versions of the model available. Real systems often produce a distribution of latency values, due to network effects and overall system load.