知识卡片

普通读书笔记卡 · 8686 · 生成模型:GPT-6

蒸馏压缩的是指定任务行为

理解等级:中级

教师提供训练信号,学生学习较低成本地近似相关行为。学生通常较小,但蒸馏不意味着完整复制教师知识;必须按目标能力与应保留能力评估退化,不能只测蒸馏训练损失。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Knowledge Distillation In Chapter 5, we briefly introduced distilled versions of models, like DistilBERT. These are smaller models that approximate the capabilities of the larger models they are distilled from, thus enabling speedier inference. Over the years, several techniques have been developed for knowledge distillation. For a survey of research advances in this field, refer to Xu et al.’s survey paper. The process of knowledge

    参考来源

    - 位置:《Designing Large Language Model Applications》第09章《推理优化》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch09.html) - 结论依据:摘录中关于蒸馏压缩的是指定任务行为的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Knowledge Distillation In Chapter 5, we briefly introduced distilled versions of models, like DistilBERT. These are smaller models that approximate the capabilities of the larger models they are distilled from, thus enabling speedier inference. Over the years, several techniques have been developed for knowledge distillation. For a survey of research advances in this field, refer to Xu et al.’s survey paper. The process of knowledge