知识卡片

普通读书笔记卡 · 6502.a.3 · 生成模型:GPT-6

终止标记不能靠随意文件名

理解等级:高级

本章根据tokenizer文件名是否含base/reasoning来选择EOS,这只在固定命名约定下成立。若移动或重命名同一文件,实际字节虽没变却可能改变停止协议。通用实现应显式保存模型变体与特殊token配置,并检查ID存在。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      #A Match HF behavior: chat model: <|im_end|>, base model: <|endoftext|> #B insert no <think> tag, just a new line Note that my Qwen3Tokenizer reimplementation in listing C.9 may appear somewhat complicated, as it aims to replicate the behavior of the official tokenizer released by the Qwen3 team in the Hugging Face Transformers library. At first glance, it appears to have a few quirks. For example,

    参考来源

    - 位置:《Build a Reasoning Model (From Scratch)》第09章《附录C Qwen3模型源码》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/appendix-c.html) - 结论依据:摘录中关于终止标记不能靠随意文件名的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:#A Match HF behavior: chat model: <|im_end|>, base model: <|endoftext|> #B insert no <think> tag, just a new line Note that my Qwen3Tokenizer reimplementation in listing C.9 may appear somewhat complicated, as it aims to replicate the behavior of the official tokenizer released by the Qwen3 team in the Hugging Face Transformers library. At first glance, it appears to have a few quirks. For example,