← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6636.f · 生成模型:GPT-6

目标密度不是训练效率全貌

理解等级:高级

一条序列中计入损失的位置比例影响监督信号,但整体效率还包括编码器、解码器、序列长度与每步计算。不能只因MLM选中位置少,就在任意预算下断言它一定更难训练或不适合部署。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Generating only the dropped tokens and not the entire sequence is computationally more efficient and saves training time. Note that unlike in Full Language Modeling, the loss is calculated over only a small proportion of tokens (the masked tokens) in the input sequence.

    参考来源

    - 位置:《Designing Large Language Model Applications》第04章《架构与学习目标》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch04.html) - 结论依据:摘录中关于目标密度不是训练效率全貌的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Generating only the dropped tokens and not the entire sequence is computationally more efficient and saves training time. Note that unlike in Full Language Modeling, the loss is calculated over only a small proportion of tokens (the masked tokens) in the input sequence.