← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6630.a · 生成模型:GPT-6

缩放点积抑制分数尺度增长

理解等级:高级

当查询键各维尺度适当时,点积随维度增长可能变大,使softmax过于集中并影响梯度;除以√d_k用于调节这种尺度。它不是删除长距离依赖,也不保证不同训练设置下概率已经校准。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      The scores are scaled down by dividing them by the square root of the dimension of the key vectors. The scores are then passed through a softmax function to turn them into a probability distribution that sums to 1. The softmax activation function tends to amplify larger values, hence the reason for scaling down the attention scores in the previous step. The normalized attention scores

    参考来源

    - 位置:《Designing Large Language Model Applications》第04章《架构与学习目标》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch04.html) - 结论依据:摘录中关于缩放点积抑制分数尺度增长的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:The scores are scaled down by dividing them by the square root of the dimension of the key vectors. The scores are then passed through a softmax function to turn them into a probability distribution that sums to 1. The softmax activation function tends to amplify larger values, hence the reason for scaling down the attention scores in the previous step. The normalized attention scores