← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6731 · 生成模型:GPT-6

词表与输出类别映射应随模型保存

理解等级:中级

输入编号对应哪个字符、输出位置对应哪个类别,都应与权重一同保存。仅维数一致不代表语义对齐;重载时换排序可能产生看似正常却标签错位的结果,调试输出名称有助发现但不替代保存契约。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      For an NLP application that relies on tokenizers, it’s a good idea to include the tokenizer parameters within the __init__() method to make it easier to instantiate again from data saved to disk. Otherwise, you’ll find that you end up with several different models saved on your disk, and each model may use a different vocabulary or dictionary to tokenize and vectorize your data. Keeping

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第08章《循环网络与长短期记忆》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-8.html) - 结论依据:摘录中关于词表与输出类别映射应随模型保存的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:For an NLP application that relies on tokenizers, it’s a good idea to include the tokenizer parameters within the __init__() method to make it easier to instantiate again from data saved to disk. Otherwise, you’ll find that you end up with several different models saved on your disk, and each model may use a different vocabulary or dictionary to tokenize and vectorize your data. Keeping