知识卡片

普通读书笔记卡 · 8678.a · 生成模型:GPT-6

句子补全损失改变训练关注点

理解等级:高级

只对句子后半段等目标位置计算损失,可强调依赖前文的完成部分。这是对训练信号位置的选择,不意味着前半段不输入,也不说明所有句尾天然更事实性;需要比较整体语言能力及事实指标。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      During pre-training, the loss is applied uniformly to all tokens generated in the sequence. However, Lee et al. observe that the tokens at the end of the sequence are more critical for factuality, since their content depends on the content that was generated prior. They propose a sentence completion loss, where the loss is calculated over only the latter part of the generated sequence.

    参考来源

    - 位置:《Designing Large Language Model Applications》第08章《对齐训练与推理》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch08.html) - 结论依据:摘录中关于句子补全损失改变训练关注点的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:During pre-training, the loss is applied uniformly to all tokens generated in the sequence. However, Lee et al. observe that the tokens at the end of the sequence are more critical for factuality, since their content depends on the content that was generated prior. They propose a sentence completion loss, where the loss is calculated over only the latter part of the generated sequence.