知识卡片
标注团队与标注质量度量
谁来标注首先是规模与公平问题:简单模型常由工程师自己用简陋工具标注,容易带偏;复杂模型则需要专职团队、第三方供应商或众包平台,规模从一人到数百人,成本甚至超过训练的计算成本。质量必须从一开始就设计进系统,常用手段有三类:一致性标注、黄金集测试题、独立 QA 抽检。质量改进更依赖管理而非惩罚:更多培训与文档、认可质量而非只看吞吐量、任务多样、工具易用、答案均衡、允许反馈。再尽责的标注员也会偶尔出错,流程要能检测并容忍偶发错误。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Feature and Training Data · 查看原书依据
Measuring Human Annotation Quality As the quality of any model is only as good as the data used to train the model, quality must be designed into the system from the start. This becomes increasingly true as the size of the annotation team grows.
参考来源
- 位置:《Reliable Machine Learning》第04章《Feature and Training Data》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch04.xhtml)
- 结论依据:原文明确阐述该主题,并给出定义、实践或反例。
- 原始内容:Measuring Human Annotation Quality As the quality of any model is only as good as the data used to train the model, quality must be designed into the system from the start. This becomes increasingly true as the size of the annotation team grows.