知识卡片
字节对训练按频次逐轮合并
BPE从字符或字节单位出发,统计当前分割中相邻token对的加权频率,合并最高频对并更新表示,直到预算或停止条件。词的出现次数必须进入统计;每轮频率也随先前合并改变。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 词表与分词
原书出处 · 1 处
- · 查看原书依据
tokens in the resulting output and their frequencies. We then construct an initial vocabulary consisting of the unique characters that make up these tokens. Starting from this initial vocabulary, we continue adding new tokens using merge rules. The merge rule is simple; we create a new token using the most frequent consecutive pairs of tokens. The merges continue until we reach the desired vocabulary size.
参考来源
- 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html)
- 结论依据:摘录中关于字节对训练按频次逐轮合并的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:tokens in the resulting output and their frequencies. We then construct an initial vocabulary consisting of the unique characters that make up these tokens. Starting from this initial vocabulary, we continue adding new tokens using merge rules. The merge rule is simple; we create a new token using the most frequent consecutive pairs of tokens. The merges continue until we reach the desired vocabulary size.