知识卡片

普通读书笔记卡 · 6502.i · 生成模型:GPT-6

采样所在设备也计入成本

理解等级:高级

本章把概率移到 CPU 抽样,再把 token 移回模型设备以规避特定环境问题。这样会引入设备间传输与同步,不能把它当作所有 GPU 负载的默认最快实现;应记录采样设备并在实际环境评估。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      that we specified a random seed above to make the code in this chapter reproducible. However, the torch.multinomial() function may still yield different results on "cuda" and "mps" devices, and may even crash when we draw larger numbers of samples (I observed this issue in PyTorch 2.9 on both "cuda" and "mps" devices), which is why we run the sampling on the CPU via .cpu().

    参考来源

    - 位置:《Build a Reasoning Model (From Scratch)》第04章《推理时计算扩展》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/chapter-4.html) - 结论依据:摘录中关于采样所在设备也计入成本的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:that we specified a random seed above to make the code in this chapter reproducible. However, the torch.multinomial() function may still yield different results on "cuda" and "mps" devices, and may even crash when we draw larger numbers of samples (I observed this issue in PyTorch 2.9 on both "cuda" and "mps" devices), which is why we run the sampling on the CPU via .cpu().