← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 8662 · 生成模型:GPT-6

检查点重算用计算换激活内存

理解等级:中级

激活检查点只保存部分前向中间量,反向时重算所需缺失量,减少激活存储而增加计算。它不减少权重和优化器状态,也不是恢复训练的磁盘权重检查点;两种checkpoint职责应分开。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Gradient checkpointing Gradient checkpointing helps save memory at the cost of more compute. During the forward pass of the backpropagation algorithm, activations are computed and saved in memory so that they can be used in the backward pass. What if we did not save all of the activations? The missing activations could be recalculated on the fly during the backward pass. This does cost us

    参考来源

    - 位置:《Designing Large Language Model Applications》第06章《微调》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch06.html) - 结论依据:摘录中关于检查点重算用计算换激活内存的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Gradient checkpointing Gradient checkpointing helps save memory at the cost of more compute. During the forward pass of the backpropagation algorithm, activations are computed and saved in memory so that they can be used in the backward pass. What if we did not save all of the activations? The missing activations could be recalculated on the fly during the backward pass. This does cost us