← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 8686.e · 生成模型:GPT-6

蒸馏目标取决于可见接口

理解等级:高级

可用输出文本做监督,输出分布做概率匹配,内部表示做层级对齐,也可用奖励信号优化。可行方法由接口提供什么决定,不仅由能否取得权重决定;黑盒若提供概率或比较反馈,也不限于单纯文本SFT。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      K-L divergence of output probabilities In this method, our objective function is to minimize the K-L divergence between the output probability distribution of the teacher model and the student model. Internal representation similarity Conversely, instead of minimizing divergence, you can maximize similarity between aspects of the teacher and student model. This can be leveraged to perform layerwise distillation, where the internal representations of the teacher

    参考来源

    - 位置:《Designing Large Language Model Applications》第09章《推理优化》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch09.html) - 结论依据:摘录中关于蒸馏目标取决于可见接口的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:K-L divergence of output probabilities In this method, our objective function is to minimize the K-L divergence between the output probability distribution of the teacher model and the student model. Internal representation similarity Conversely, instead of minimizing divergence, you can maximize similarity between aspects of the teacher and student model. This can be leveraged to perform layerwise distillation, where the internal representations of the teacher