知识卡片

普通读书笔记卡 · 8948 · 生成模型:GPT-6

因果约束必须跨层成立

理解等级:高级

多层网络只有在各层所有信息路径均遵守前缀可见性时,才保持最终预测的因果性。注意力掩码、目标对齐、词元内处理及残差都需一起检查。若某层已混入未来内容,后续再加掩码不能消除表示中已经携带的泄漏。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      This also works for transformers with more than one layer, where the masking strategy looks like shown in figure 26.19. Figure 26.19: Multilayer masked attention achieves causal prediction with a deep net. Notice that the output tokens on every layer l have the property that only depends on , where T0 are the initial set of tokens input into the transformer. Also notice that, after

    参考来源

    - 位置:《Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)》第26章《Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)/_epub-src/OPS/xhtml/c026.xhtml) - 结论依据:摘录中关于因果约束必须跨层成立的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:This also works for transformers with more than one layer, where the masking strategy looks like shown in figure 26.19. Figure 26.19: Multilayer masked attention achieves causal prediction with a deep net. Notice that the output tokens on every layer l have the property that only depends on , where T0 are the initial set of tokens input into the transformer. Also notice that, after