← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6654.c · 生成模型:GPT-6

停用词与稀有词过滤会改变证据

理解等级:中级

过滤某些高频或低频词能减少维数与噪声,也可能移除否定、实体名或关键专业词。应根据任务与数据评估取舍,并保存词表及过滤规则;少量出现不等于不重要,常见也不等于毫无语义。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Here’s how our token sorter fits into an NLP pipeline right after a tokenizer (see chapter 2). We have included a stop word filter as well as a rare word filter in our mechanical token sorter sketch illustrated in figure 1.6. Strings flow in from the top, and BOW vectors are created from the height profile of the token “stacks” at the bottom.

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第01章《会读写的机器与自然语言处理》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-1.html) - 结论依据:摘录中关于停用词与稀有词过滤会改变证据的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Here’s how our token sorter fits into an NLP pipeline right after a tokenizer (see chapter 2). We have included a stop word filter as well as a rare word filter in our mechanical token sorter sketch illustrated in figure 1.6. Strings flow in from the top, and BOW vectors are created from the height profile of the token “stacks” at the bottom.