知识卡片
简化训练更新链路
结构图:
flowchart LR
Q[抽取题目] --> R[无梯度采样多回答]
R --> W[验证终值及格式]
W --> A[计算组内优势]
R --> P[有梯度重算回答对数概率]
A --> L[优势加权策略损失]
P --> L
L --> B[反向与梯度裁剪]
B --> U[优化器更新]
U --> Q
记录奖励、长度和检查点,并在独立评估集检验结果。本图对应本章简化实现,未画入原始 GRPO 的概率比裁剪和 KL。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- · 查看原书依据
with verifiable rewards using the GRPO algorithm, as illustrated in figure 6.19. Figure 6.19 After implementing the individual GRPO stages, we now implement the surrounding training loop to update the model weights. The training loop consists of eight main stages, as shown in figure 6.20. Most of these stages are standard components of a typical PyTorch training loop used for deep neural networks, including LLMs.
参考来源
- 位置:《Build a Reasoning Model (From Scratch)》第06章《强化学习训练推理模型》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/chapter-6.html)
- 结论依据:摘录中关于简化训练更新链路的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:with verifiable rewards using the GRPO algorithm, as illustrated in figure 6.19. Figure 6.19 After implementing the individual GRPO stages, we now implement the surrounding training loop to update the model weights. The training loop consists of eight main stages, as shown in figure 6.20. Most of these stages are standard components of a typical PyTorch training loop used for deep neural networks, including LLMs.