知识卡片

普通读书笔记卡 · 6631.a · 生成模型:GPT-6

线性偏置与旋转位置不同

理解等级:高级

ALiBi在注意力分数加入与距离相关的线性偏置;RoPE对查询键按位置旋转。两者改变位置依赖的方式不同,RoPE的具体权重不必随距离逐点单调下降,二者也不无条件保证无限长度外推。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Attention with Linear Biases (ALiBi) In this technique, the attention scores are penalized with a bias term proportional to the distance between the query token and the key token. This technique also enables modeling sequences of longer length during inference than what was encountered in the training process. Rotary Position Embedding (RoPE) Just like ALiBi, this technique has the property of relative decay; there is

    参考来源

    - 位置:《Designing Large Language Model Applications》第04章《架构与学习目标》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch04.html) - 结论依据:摘录中关于线性偏置与旋转位置不同的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Attention with Linear Biases (ALiBi) In this technique, the attention scores are penalized with a bias term proportional to the distance between the query token and the key token. This technique also enables modeling sequences of longer length during inference than what was encountered in the training process. Rotary Position Embedding (RoPE) Just like ALiBi, this technique has the property of relative decay; there is