← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 8942.b · 生成模型:GPT-6

查询键维度必须匹配

理解等级:中级

查询q与键k需要具有相同维度m才能计算内积;值向量可以有不同维度dv。注意力输出维度由值决定,若要与d维原词元作残差相加,需通过输出映射恢复到d。不要将示意图中相同字母误作所有中间维度必须相等。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      We define these vectors as linear transformations of the token's code vector, projecting to query/key/value vectors of length m. For a token t, we have: (26.11) (26.12) (26.13) Here is a question to think about: Could you use other differentiable functions to compute the query, value, and key? Would that be useful? In transformers, all inputs to the net are tokenized, so the textual question

    参考来源

    - 位置:《Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)》第26章《Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)/_epub-src/OPS/xhtml/c026.xhtml) - 结论依据:摘录中关于查询键维度必须匹配的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:We define these vectors as linear transformations of the token's code vector, projecting to query/key/value vectors of length m. For a token t, we have: (26.11) (26.12) (26.13) Here is a question to think about: Could you use other differentiable functions to compute the query, value, and key? Would that be useful? In transformers, all inputs to the net are tokenized, so the textual question