知识卡片
未知类型占比不同于未知词元占比
新出现2546个词表类型说明领域覆盖变化,但不能据此断言每段文本约10%的词元都未知。少见类型可有低出现量,应另外统计未知词元频次和文档分布;跨域下降也可能来自已见词含义及类别先验变化。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 思维的词元
原书出处 · 1 处
- · 查看原书依据
The combined DataFrame of BOWs has tokens that were used in product reviews but not movie reviews. You now have 23,302 unique tokens for both movie reviews and products in your vocabulary. Movie reviews only contained 20,756 unique tokens, so there must be 23,302 – 20,756, or 2,546, new tokens about products that weren’t previously in your vocabulary.
参考来源
- 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html)
- 结论依据:摘录中关于未知类型占比不同于未知词元占比的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:The combined DataFrame of BOWs has tokens that were used in product reviews but not movie reviews. You now have 23,302 unique tokens for both movie reviews and products in your vocabulary. Movie reviews only contained 20,756 unique tokens, so there must be 23,302 – 20,756, or 2,546, new tokens about products that weren’t previously in your vocabulary.