知识卡片
点积分数经缩放再归一
每个查询与候选键计算点积分数sⱼ=qᵀkⱼ,再对sⱼ/√m作Softmax。缩放发生在指数归一化之前,用来控制随查询键维度变化的分数尺度。对归一化结果再除√m既不等价,也会破坏权重和为1的性质。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- · 查看原书依据
This token will submit its query vector, qquestion to be matched against the keys of the tokens that represent different patches in the image; the similarity between the query and the key determines the amount of attention weight the query will apply to the token with that key. The most common measure of similarity between a query q and a key k is the dot
参考来源
- 位置:《Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)》第26章《Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)/_epub-src/OPS/xhtml/c026.xhtml)
- 结论依据:摘录中关于点积分数经缩放再归一的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:This token will submit its query vector, qquestion to be matched against the keys of the tokens that represent different patches in the image; the similarity between the query and the key determines the amount of attention weight the query will apply to the token with that key. The most common measure of similarity between a query q and a key k is the dot