知识卡片
多数故障不是 ML 故障
训练系统是「恰好对所处理数据极度敏感」的复杂数据处理流水线,通常还分布在多台机器上,这种底子天然不利于长期可靠;生产工程师也常盯着数据敏感性找最常见故障。但有经验的从业者回看 ML 系统历年事故,发现多数故障并非 ML 特有,而是这类分布式系统里常见的软件与系统故障——其影响与检测带有 ML 特征,根因却普通。作者举了两个略带幽默的例子:训练系统失去读取数据权限、于是模型在空数据上训练;同步到服务的版本并不是我们以为的那个版本。因此要让训练系统可靠,先缓解系统与软件错误:做好软件版本与部署、权限与数据访问要求、数据更新策略与组织、复制系统与校验,把通用分布式系统的可靠性工作做完,再开始任何 ML 特有工作。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Training Systems · 查看原书依据
However, when experienced practitioners look at the experienced failures in ML systems over time, they find that most of the failures are not ML specific.4 They are software and systems failures that occur commonly in this kind of distributed system. These failures often have impact and detection challenges that are ML specific, but the underlying causes are most commonly not ML specific.
参考来源
- 位置:《Reliable Machine Learning》第07章《Training Systems》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch07.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:However, when experienced practitioners look at the experienced failures in ML systems over time, they find that most of the failures are not ML specific.4 They are software and systems failures that occur commonly in this kind of distributed system. These failures often have impact and detection challenges that are ML specific, but the underlying causes are most commonly not ML specific.