知识卡片
字符覆盖不等于任意输入无未知项
字符级子词若保留所需字符,可把未见整词拆为片段,但新字符、长度限制或实现规则仍可产生未知项。字节基础覆盖与字符基础覆盖不同,不能把所有子词算法都说成从不需要UNK,也不能由能编码推断理解其含义。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 思维的词元
原书出处 · 1 处
- · 查看原书依据
Many of these character sequences will be whole words or even compound words, but many will be pieces of words. In fact, all subword tokenizers maintain a token within the vocabulary for every individual character in your vocabulary. This means it never needs to use an OOV token, as long as any new text doesn’t contain any new characters. Subword tokenizers attempt to optimally clump
参考来源
- 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html)
- 结论依据:摘录中关于字符覆盖不等于任意输入无未知项的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Many of these character sequences will be whole words or even compound words, but many will be pieces of words. In fact, all subword tokenizers maintain a token within the vocabulary for every individual character in your vocabulary. This means it never needs to use an OOV token, as long as any new text doesn’t contain any new characters. Subword tokenizers attempt to optimally clump