← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 8679.b · 生成模型:GPT-6

回看特征需按原定义计算

理解等级:高级

Lookback Lens比较上下文与已生成token的平均注意力,再用 A_context/(A_context+A_new)归一化;按层和头收集特征。它不是直接相除两边注意力总量,注意力也只是检测特征,不是事实真实性的因果证明。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      lookback ratio, defined as the ratio of the attention weights on the context tokens to the attention weights on the newly generated tokens. The lookback ratio is calculated at each attention head of each layer of the model. These ratios are used as features to train a linear classifier model.

    参考来源

    - 位置:《Designing Large Language Model Applications》第08章《对齐训练与推理》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch08.html) - 结论依据:摘录中关于回看特征需按原定义计算的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:lookback ratio, defined as the ratio of the attention weights on the context tokens to the attention weights on the newly generated tokens. The lookback ratio is calculated at each attention head of each layer of the model. These ratios are used as features to train a linear classifier model.