知识卡片
免子词模型仍有输入单位
所谓tokenization-free通常移除外部学习子词切分,仍将字符码点或字节映射成数值输入。CANINE、ByT5等选择不同粒度或内部压缩,不能理解成网络直接接收没有任何表示约定的原始文字。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 词表与分词
原书出处 · 1 处
- · 查看原书依据
There have been forays into the world of tokenization-free language modeling, with models like CANINE, ByT5, and Charformer. CANINE accepts Unicode codepoints as input. But there are 1,114,112 possible code points, rendering the vocabulary and resulting embedding layer size infeasible. To resolve this, CANINE uses hashed embeddings so that the effective vocabulary space is much smaller. ByT5 accepts input in terms of bytes, so there
参考来源
- 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html)
- 结论依据:摘录中关于免子词模型仍有输入单位的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:There have been forays into the world of tokenization-free language modeling, with models like CANINE, ByT5, and Charformer. CANINE accepts Unicode codepoints as input. But there are 1,114,112 possible code points, rendering the vocabulary and resulting embedding layer size infeasible. To resolve this, CANINE uses hashed embeddings so that the effective vocabulary space is much smaller. ByT5 accepts input in terms of bytes, so there