知识卡片

普通读书笔记卡 · 6501.l · 生成模型:GPT-6

残差路径要求投影回原宽度

理解等级:中级

本章块先归一化再注意力,并加回原输入;随后再次归一化、前馈并加回中间残差。注意力和前馈内部宽度可变化,但加法两侧形状必须相容。残差并非拼接,也不表示跳过子层计算。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      x = x + shortcut shortcut = x x = self.norm2(x) x = self.ff(x) x = x + shortcut return x, next_cache #A The shape is (batch_size, num_tokens, emb_size) As we can see, in listing C.5, the transformer block simply connects various elements we implemented in previous sections. C.6 Main model code In this section, we will define the Qwen3Model class that we imported and

    参考来源

    - 位置:《Build a Reasoning Model (From Scratch)》第09章《附录C Qwen3模型源码》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/appendix-c.html) - 结论依据:摘录中关于残差路径要求投影回原宽度的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:x = x + shortcut shortcut = x x = self.norm2(x) x = self.ff(x) x = x + shortcut return x, next_cache #A The shape is (batch_size, num_tokens, emb_size) As we can see, in listing C.5, the transformer block simply connects various elements we implemented in previous sections. C.6 Main model code In this section, we will define the Qwen3Model class that we imported and