知识卡片
词表大小存在资源权衡
更大的词表可能减少某段文本的切分长度,同时扩大嵌入和输出投影等与词表关联的开销。不能仅凭词表更大推断总体更快;需同时比较输入分布、序列长度、模型维度和解码实现。书中“token 翻倍成本大致翻倍”只宜作粗略直觉。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- · 查看原书依据
A larger vocabulary in a language model increases its size and computational cost for each individually generated token, but it also allows more words to be represented as single tokens rather than being split into subword components. This is beneficial because splitting a word (like breaking "Explain" into "Ex" and "plain") results in more input tokens. More tokens lead to longer input sequences, which increases
参考来源
- 位置:《Build a Reasoning Model (From Scratch)》第02章《使用预训练模型生成文本》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/chapter-2.html)
- 结论依据:摘录中关于词表大小存在资源权衡的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:A larger vocabulary in a language model increases its size and computational cost for each individually generated token, but it also allows more words to be represented as single tokens rather than being split into subword components. This is beneficial because splitting a word (like breaking "Explain" into "Ex" and "plain") results in more input tokens. More tokens lead to longer input sequences, which increases