知识卡片

普通读书笔记卡 · 6633 · 生成模型:GPT-6

独热交叉熵只取目标项

理解等级:中级

真实下一token用独热标签表示时,交叉熵为−log p(target),其他项标签为零而不贡献。这衡量模型给观察到目标分配多少概率,不说明语境中只有一个合法续写;训练仍会受语料实际选择影响。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Cross-Entropy = -log(output probability of gold truth token) Once the loss is calculated, the gradient of the loss with respect to the parameters of the model is calculated and the weights are updated, using the backpropagation algorithm. Intrinsic Model Evaluation How do we know if the backpropagation algorithm is actually working and that the model is getting better over time? We can use either intrinsic

    参考来源

    - 位置:《Designing Large Language Model Applications》第04章《架构与学习目标》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch04.html) - 结论依据:摘录中关于独热交叉熵只取目标项的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Cross-Entropy = -log(output probability of gold truth token) Once the loss is calculated, the gradient of the loss with respect to the parameters of the model is calculated and the weights are updated, using the backpropagation algorithm. Intrinsic Model Evaluation How do we know if the backpropagation algorithm is actually working and that the model is getting better over time? We can use either intrinsic