知识卡片

普通读书笔记卡 · 6511.b · 生成模型:GPT-6

标准差口径影响优势尺度

理解等级:高级

优势归一化需要记录样本或总体标准差的口径。书中 torch.std 默认使用样本校正,使 [1,1,0,0] 的优势约为 ±.866 而非 ±1;ε 防零分母,单元素组仍需另行处理,不能仅靠加 ε 修复未定义标准差。补充依据:PyTorch std。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4) print(rewards) print(advantages) The rewards, which we computed in the previous section, are [1., 1., 0., 0.], and the corresponding advantages are [0.8659, 0.8659, -0.8659, -0.8659]. These advantage values capture which responses performed better than average and which performed worse than average within the group. Note The "GR" (group relative) in GRPO refers to the fact that

    参考来源

    - 位置:《Build a Reasoning Model (From Scratch)》第06章《强化学习训练推理模型》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/chapter-6.html) - 结论依据:摘录中关于标准差口径影响优势尺度的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4) print(rewards) print(advantages) The rewards, which we computed in the previous section, are [1., 1., 0., 0.], and the corresponding advantages are [0.8659, 0.8659, -0.8659, -0.8659]. These advantage values capture which responses performed better than average and which performed worse than average within the group. Note The "GR" (group relative) in GRPO refers to the fact that