知识卡片

普通读书笔记卡 · 7003.f · 生成模型:DeepSeek-V4.1-Flash

模型质量与性能评估

理解等级:中级

评估要先定义“什么是好(working)”并据此度量,通常跨不同子集(slices)测量效果。流程:先离线用代表性查询比对“正确”答案建立基本信心;再整体或暗启动(dark launch)集成;最后才正式发布。定义与度量本身极具挑战,因为要先决定衡量什么,详见第 5 章。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Introduction · 查看原书依据

      It turns out that surprisingly detailed work is required to actually answer that question—beginning with the almost amusing, but absolutely true point that we have to decide what will count as working, and how we will evaluate model performance against that target. This usually involves identifying the effect that we’re trying to create, and measuring it across various subsets (or slices) of representative queries or use cases

    参考来源

    - 位置:《Reliable Machine Learning》第01章《Introduction》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch01.xhtml) - 结论依据:原文指出评估的难点先在于“决定什么算 working”,再谈如何度量。 - 原始内容:It turns out that surprisingly detailed work is required to actually answer that question—beginning with the almost amusing, but absolutely true point that we have to decide what will count as working, and how we will evaluate model performance against that target. This usually involves identifying the effect that we’re trying to create, and measuring it across various subsets (or slices) of representative queries or use cases