← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6615.b · 生成模型:GPT-6

去重可降低记忆但不是清除保证

理解等级:中级

减少重复片段可能降低逐字记忆和训练验证重叠,并节约某些冗余计算。但原研究中的改善倍数不是所有模型都适用;单次出现也可能被记住,不能把去重完成当作不泄漏证明。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      and thwarts the model’s ability to generalize. While there are many ways to quantify memorization, we will focus on memorization by generation, where a model is said to have memorized a sequence if it is capable of generating it verbatim. Lee et al. have shown that models trained on datasets that have been deduplicated at the sequence level generate ten times less verbatim training data.

    参考来源

    - 位置:《Designing Large Language Model Applications》第02章《预训练数据》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch02.html) - 结论依据:摘录中关于去重可降低记忆但不是清除保证的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:and thwarts the model’s ability to generalize. While there are many ways to quantify memorization, we will focus on memorization by generation, where a model is said to have memorized a sequence if it is capable of generating it verbatim. Lee et al. have shown that models trained on datasets that have been deduplicated at the sequence level generate ten times less verbatim training data.