← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6656.g · 生成模型:GPT-6

分词器必须随模型保存并固定

理解等级:中级

训练模型的嵌入行和输出编号对应固定词表,换一个分词器即使输出仍是整数,也可能改变编号含义。应共同保存词表、合并或分段规则、特殊项和归一化设置;使用预训练模型通常先复用匹配分词器,不是每换一份语料都必须重训。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Another challenge of BPE tokenizers is all the bookkeeping you need to do to keep track of which trained tokenizer goes with each of your trained models. Addressing this problem was one of Hugging Face’s major innovations. Their team made it easy to store and share all the preprocessing data, such as the tokenizer vocabulary, alongside the language model. This makes it easier to reuse

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html) - 结论依据:摘录中关于分词器必须随模型保存并固定的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Another challenge of BPE tokenizers is all the bookkeeping you need to do to keep track of which trained tokenizer goes with each of your trained models. Addressing this problem was one of Hugging Face’s major innovations. Their team made it easy to store and share all the preprocessing data, such as the tokenizer vocabulary, alongside the language model. This makes it easier to reuse