知识卡片

普通读书笔记卡 · 6748.c · 生成模型:GPT-6

损失忽略填充与注意力屏蔽不同

理解等级:中级

CrossEntropyLoss的ignore_index仅让对应标签不参与目标归约,不能阻止模型读取输入padding。还要配置注意力mask,并按有效目标token统计损失;TRG_PAD_IDX必须来自实际目标分词器。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      >>> criterion = nn.CrossEntropyLoss(ignore_index=TRG_PAD_IDX) #1 #1 Ignores padding in the input gradient calculation Ben Trevett contributed much of the code for the PyTorch Transformer beginner tutorial. He, along with colleagues, has written an outstanding and informative Jupyter Notebook series for their PyTorch Seq2Seq tutorial,25 covering sequence-to-sequence models. Their “Attention Is All You Need” notebook26 provides a from-scratch implementation of a basic transformer model. To avoid

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第09章《可堆叠的Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-9.html) - 结论依据:摘录中关于损失忽略填充与注意力屏蔽不同的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:>>> criterion = nn.CrossEntropyLoss(ignore_index=TRG_PAD_IDX) #1 #1 Ignores padding in the input gradient calculation Ben Trevett contributed much of the code for the PyTorch Transformer beginner tutorial. He, along with colleagues, has written an outstanding and informative Jupyter Notebook series for their PyTorch Seq2Seq tutorial,25 covering sequence-to-sequence models. Their “Attention Is All You Need” notebook26 provides a from-scratch implementation of a basic transformer model. To avoid