← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 8945 · 生成模型:GPT-6

预归一化块保留残差通路

理解等级:中级

书中ViT块先计算T′=T+MSA(LN(T)),再计算T″=T′+MLP(LN(T′))。两个残差分支分别增加跨位置汇集和逐词元变换,并保留原表示的直接路径。归一化位置属于结构选择,不能把该写法当作所有Transformer唯一布局。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Beyond this basic template, there are many variations that can be added, resulting in different particular architectures within the transformer family. Some common additions are normalization layers and residual connections. In figure 26.14 we plot the ViT architecture from [109], showing where these additional pieces enter the picture.

    参考来源

    - 位置:《Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)》第26章《Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)/_epub-src/OPS/xhtml/c026.xhtml) - 结论依据:摘录中关于预归一化块保留残差通路的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Beyond this basic template, there are many variations that can be added, resulting in different particular architectures within the transformer family. Some common additions are normalization layers and residual connections. In figure 26.14 we plot the ViT architecture from [109], showing where these additional pieces enter the picture.