知识卡片
统计偏差与校准图
金丝雀指标善于提示“出了大问题”,却分不出好与更好,正常也不代表万事大吉。统计偏差(统计义而非伦理义)比较“按预测期望看到的总量”与“实际看到的总量”是否相等,理想值为 0.00,偏离几个百分点往往就是警报;它常与切片分析搭配定位问题。但零偏差可被轻易伪造:一个对每个样本都返回观测均值的模型偏差为零却毫无信息量,所以必须看细粒度切片。校准图按模型分数分桶、逐桶算偏差,画观测发生率对预测概率,能看出系统性高估低估及输出两端是否可靠。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Evaluating Model Validity and Quality · 查看原书依据
The drawback of bias as a metric is that it is trivial to create a model with perfectly zero bias, but that is a terrible model. As a thought experiment, this could be done by having a model that just returns the average observed value for every example—zero bias in aggregate, but totally uninformative.
参考来源
- 位置:《Reliable Machine Learning》第05章《Evaluating Model Validity and Quality》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch05.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:The drawback of bias as a metric is that it is trivial to create a model with perfectly zero bias, but that is a terrible model. As a thought experiment, this could be done by having a model that just returns the average observed value for every example—zero bias in aggregate, but totally uninformative.