知识卡片

普通读书笔记卡 · 7017.l · 生成模型:DeepSeek-V4.1-Flash

停止反复推送,让模型 roll forward

理解等级:高级

issue B 修好后 App 团队再次推送,又引发一轮设备更新与又一轮重复查询;此时 ops 已按下持续模型训练的停止键,于是「最新版本」恰好是被污染数据影响过的那个。消息传开后根因虽追到最近的 App 推送,但「App 更新为何改变模型预测」并不显然——「更新会导致重复查询」这一行为并未广为人知,App 侧知情者也没把它与持续模型的训练数据联系起来;于是大家猜测更新里另有 bug,决定把再更新回滚、观察几小时,而这次再回滚又制造了更多重复查询与污染数据。等回滚完成、观察数小时后,信息终于足够确认问题只出在「推送的做法」上。缓解出人意料地简单:停止对 App 做回滚与再更新,让持续 ML 系统自行 roll forward 追上世界的新状态;模型最终看到干净数据、点击率回到正常。要点极精炼:许多「尝试中的缓解动作」造成的伤害与好处相当,某种程度上还延长了影响期;回头看,若一开始就让模型滚过去,系统很可能比经历这一通抢修恢复得更平滑、更体面——有时候就是该稳住、让它滚过去。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Case Studies: MLOps in Practice · 查看原书依据

      In retrospect, if we had just allowed the model to roll through, the system likely would have recovered much more smoothly and gracefully than it did in the wake of all our attempts to fix things. Sometimes it pays to just hold tight and roll through.

    参考来源

    - 位置:《Reliable Machine Learning》第15章《Case Studies: MLOps in Practice》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch15.xhtml) - 结论依据:原文明确阐述该主题,并给出定义、实践或反例。 - 原始内容:In retrospect, if we had just allowed the model to roll through, the system likely would have recovered much more smoothly and gracefully than it did in the wake of all our attempts to fix things. Sometimes it pays to just hold tight and roll through.