知识卡片

普通读书笔记卡 · 7013.h · 生成模型:DeepSeek-V4.1-Flash

日志馈送进程内存耗尽:静默停摆三周

理解等级:中级

顺着训练系统下钻:调度器把状态存储进程与读取前一日搜索日志、更新模型的进程分开;Ariel 发现所有读日志的进程大部分时间都在等日志返回,而负责从日志系统读取原始客户日志的 10 个 log feeder 进程每隔几分钟就因内存分配失败崩溃退出。崩溃后新进程在新机器上拉起、训练进程重连、读几个字节、再次内存耗尽崩溃——这个循环已经持续三周,正是训练跑不完、模型停更的近因。把 feeder 从 10 增到 20 分摊负载(必要时加内存)后崩溃停止,几小时后训练完成、新模型自动加载进 serving,前五条点击率回到应有水平。作者借这段提醒:ML 故障常是分布式管线在某一扩容阈值上性能骤降,而触发器只是数据的自然增长;而且此时只有近因、没有明显缓解手段——没有新的训练模型,就无法改善现状,缓解与解决往往合并为同一步。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Incident Response · 查看原书依据

      Diving into process crash logs, Ariel sees that the log feeders are running out of memory, and when they can’t allocate more memory, they crash. When they crash, a new log-feeder process is started on a new machine, and the training process retries its connection, reads a few bytes, and then that process runs out of memory and crashes again.

    参考来源

    - 位置:《Reliable Machine Learning》第11章《Incident Response》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch11.xhtml) - 结论依据:原文明确阐述该主题,并给出定义、实践或反例。 - 原始内容:Diving into process crash logs, Ariel sees that the log feeders are running out of memory, and when they can’t allocate more memory, they crash. When they crash, a new log-feeder process is started on a new machine, and the training process retries its connection, reads a few bytes, and then that process runs out of memory and crashes again.