知识卡片
日志馈送进程内存耗尽:静默停摆三周
顺着训练系统下钻:调度器把状态存储进程与读取前一日搜索日志、更新模型的进程分开;Ariel 发现所有读日志的进程大部分时间都在等日志返回,而负责从日志系统读取原始客户日志的 10 个 log feeder 进程每隔几分钟就因内存分配失败崩溃退出。崩溃后新进程在新机器上拉起、训练进程重连、读几个字节、再次内存耗尽崩溃——这个循环已经持续三周,正是训练跑不完、模型停更的近因。把 feeder 从 10 增到 20 分摊负载(必要时加内存)后崩溃停止,几小时后训练完成、新模型自动加载进 serving,前五条点击率回到应有水平。作者借这段提醒:ML 故障常是分布式管线在某一扩容阈值上性能骤降,而触发器只是数据的自然增长;而且此时只有近因、没有明显缓解手段——没有新的训练模型,就无法改善现状,缓解与解决往往合并为同一步。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Incident Response · 查看原书依据
Diving into process crash logs, Ariel sees that the log feeders are running out of memory, and when they can’t allocate more memory, they crash. When they crash, a new log-feeder process is started on a new machine, and the training process retries its connection, reads a few bytes, and then that process runs out of memory and crashes again.
参考来源
- 位置:《Reliable Machine Learning》第11章《Incident Response》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch11.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:Diving into process crash logs, Ariel sees that the log feeders are running out of memory, and when they can’t allocate more memory, they crash. When they crash, a new log-feeder process is started on a new machine, and the training process retries its connection, reads a few bytes, and then that process runs out of memory and crashes again.