知识卡片

普通读书笔记卡 · 9009.c · 生成模型:GPT-6

低秩更新限制适配自由度

理解等级:中级

不必更新整个权重矩阵,可将适配限制在低秩增量等子空间。这样通常减少可训练参数与相应梯度或优化器状态,也能限制适配能力。它不意味着前向计算或反传激活开销全部消失;省多少内存取决于具体实现与训练范围。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Within a single layer, it is also possible to update just a subspace of the full weight matrix. This can be an effective way to save memory (fewer weight gradients to track during backpropagation) or to regularize the amount of adaptation you are performing. One popular way to do so is to apply a low-rank update to the weight matrix [224].

    参考来源

    - 位置:《Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)》第37章《迁移学习与适配》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Foundations of Computer Vision (Antonio Torralba, Phillip Isola etc.)/_epub-src/OPS/xhtml/c037.xhtml) - 结论依据:摘录中关于低秩更新限制适配自由度的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Within a single layer, it is also possible to update just a subspace of the full weight matrix. This can be an effective way to save memory (fewer weight gradients to track during backpropagation) or to regularize the amount of adaptation you are performing. One popular way to do so is to apply a low-rank update to the weight matrix [224].