知识卡片
生育率依赖词数定义
token fertility用token数除以词数,描述某数据在某分词器下的平均切分程度。词数定义、语言和特殊token计入范围会影响结果;它是压缩指标,不能单独推导语义理解或下游任务质量。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 词表与分词
原书出处 · 1 处
- · 查看原书依据
Fertility is a measure of the average number of tokens needed to represent a dataset. It is calculated by dividing the number of tokens in a dataset by the number of words in a dataset. The higher the fertility, the lower the compression power of the tokenizer. Goldman et al. show that higher compression leads to better downstream performance, although this is disputed in experiments
参考来源
- 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html)
- 结论依据:摘录中关于生育率依赖词数定义的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Fertility is a measure of the average number of tokens needed to represent a dataset. It is calculated by dividing the number of tokens in a dataset by the number of words in a dataset. The higher the fertility, the lower the compression power of the tokenizer. Goldman et al. show that higher compression leads to better downstream performance, although this is disputed in experiments