知识卡片
黄金数据集
持续重训并用渐进式验证的系统里,很难分清是模型变差了,还是数据本身变得更容易或更难预测。黄金数据集是解法:取某个固定时间点(如去年 10 月 1 日)的一批数据,明令永不用于训练,并连同当时的模型输出一起封存,甚至把这些结果视为“正确”。由于新训练数据只会让模型在黄金集上表现缓慢变化,一旦指标突变就极有利于定位 bug。局限也明确:它不适合判断绝对质量,相关性随时间衰减;数据隐私法规或删除、过期要求也可能让它无法长期保留。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Evaluating Model Validity and Quality · 查看原书依据
One way to control this is to create a golden set of data that models are not ever allowed to train on, but that is from a specific point in time. For example, we might decide that the data from October 1 of last year might be set aside as golden set data, never to be used for training under any circumstance, but held aside.
参考来源
- 位置:《Reliable Machine Learning》第05章《Evaluating Model Validity and Quality》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch05.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:One way to control this is to create a golden set of data that models are not ever allowed to train on, but that is from a specific point in time. For example, we might decide that the data from October 1 of last year might be set aside as golden set data, never to be used for training under any circumstance, but held aside.