知识卡片
训练过快与竞态发散
生产工程师有时会惊讶地发现模型能「训练得太快」:取决于算法、结构与并行度,同一模型训得快时产出垃圾、训得慢时却准确得多。机制是模型状态由一组分布式学习进程共享,各进程读新数据、查当前状态、再更新状态的一部分;只要中间有锁(慢)或恰好没人更新同一键(大规模下不太可能)就没事,但多个 learner 同时查询、排队并更新就会堆叠,使该部分模型朝某方向移动过远,下一轮 learner 又发现它严重偏斜而朝反方向大幅修正,久而久之模型发散而非收敛。对分布式训练,多重竞态是极常见的故障源,却没有简单办法判定是否「过快」;唯一现实(且恼人)的检验是把同一模型用更多/更少 learner 各训一遍,若慢的那个在某些质量指标上更好,就可能过快。缓解方向是结构性地限制在途更新数量:把状态放进 RAM 等极快存储、限制多进程更新速率;对每键加锁虽可行但性能代价通常高到不值得。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Training Systems · 查看原书依据
But it is entirely possible to have a model that, when trained too quickly, produces garbage results, but when trained more slowly, produces much more accurate results. Here is one way this can happen: there is a distributed representation of the state of the model used by a distributed set of learning processes.
参考来源
- 位置:《Reliable Machine Learning》第07章《Training Systems》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch07.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:But it is entirely possible to have a model that, when trained too quickly, produces garbage results, but when trained more slowly, produces much more accurate results. Here is one way this can happen: there is a distributed representation of the state of the model used by a distributed set of learning processes.