← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6743.c · 生成模型:GPT-6

多头并行形成不同表示子空间

理解等级:中级

各头有独立Q、K、V投影,分别聚合后拼接,再经输出投影回目标宽度。固定总宽度时每头通常更窄,头数增加不自动提高容量或质量;某头也不一定对应一个可命名词义或语法功能。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Multi-head self-attention Multi-head self-attention is an expansion of the self-attention approach to creating multiple attention heads that each attend to different aspects of the words in a text. So if a token has multiple meanings that are all relevant to the interpretation of the input text, they can each be accounted for in the separate attention heads. You can think of each attention head as

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第09章《可堆叠的Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-9.html) - 结论依据:摘录中关于多头并行形成不同表示子空间的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Multi-head self-attention Multi-head self-attention is an expansion of the self-attention approach to creating multiple attention heads that each attend to different aspects of the words in a text. So if a token has multiple meanings that are all relevant to the interpretation of the input text, they can each be accounted for in the separate attention heads. You can think of each attention head as