← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6498.b.8 · 生成模型:GPT-6

奖励塑形需要明确各分量

理解等级:中级

可为正确但缺格式的回答给部分奖励,以提供不同梯度信号;但各分量与权重改变了优化目标。应防止模型仅靠格式获得高分,并以独立终值正确率和格式合规率检查塑形效果,而不只最大化训练奖励。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Exercise 6.1: Adding format-aware reward shaping Extend the reward_rlvr function from this chapter so that it assigns partial credit based on output format. Specifically, modify the reward function so that it returns 1.0 if the model produces the correct answer in the required \boxed{} format, 0.5 if the answer is correct but not boxed, and 0.0 otherwise. 6.7 Preparing learning signals from rollouts via advantages

    参考来源

    - 位置:《Build a Reasoning Model (From Scratch)》第06章《强化学习训练推理模型》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/chapter-6.html) - 结论依据:摘录中关于奖励塑形需要明确各分量的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Exercise 6.1: Adding format-aware reward shaping Extend the reward_rlvr function from this chapter so that it assigns partial credit based on output format. Specifically, modify the reward function so that it returns 1.0 if the model produces the correct answer in the required \boxed{} format, 0.5 if the answer is correct but not boxed, and 0.0 otherwise. 6.7 Preparing learning signals from rollouts via advantages