知识卡片

普通读书笔记卡 · 8670.d · 生成模型:GPT-6

子集微调保留参数也限制容量

理解等级:中级

可冻结低层、只训练高层或偏置,以减少更新参数。目标效果取决于模型是否具备相关参数及任务是否能由该子集适配;层级功能不是绝对分工,也不能从某实验接近全参推出普遍等价。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Subset Methods A naive way of choosing a subset of parameters to fine-tune on would be to fine-tune only the upper layers of the Transformer and keep everything else frozen. The lower layers of the Transformer are known to be specialized in more fundamental aspects of language like syntax, which we want the LLM to preserve. Another way is to fine-tune only the bias terms

    参考来源

    - 位置:《Designing Large Language Model Applications》第07章《进阶微调技术》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch07.html) - 结论依据:摘录中关于子集微调保留参数也限制容量的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Subset Methods A naive way of choosing a subset of parameters to fine-tune on would be to fine-tune only the upper layers of the Transformer and keep everything else frozen. The lower layers of the Transformer are known to be specialized in more fundamental aspects of language like syntax, which we want the LLM to preserve. Another way is to fine-tune only the bias terms