知识卡片

普通读书笔记卡 · 6673 · 生成模型:GPT-6

词面归一化不能合并全部同义表达

理解等级:中级

词干和词形还原主要合并部分形式变体,beautiful与pretty等不同拼写的近义词仍可能占不同词表坐标。反过来,形式近似或归一化相同也可能含义不同;需要借助语境和任务评价,而非把拼写接近视为语义接近。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      TF–IDF vectors count the terms according to their exact spelling in a document. So texts that restate the same meaning will have completely different TF–IDF vector representations if they spell things differently or use different words. This messes up search engines and document similarity comparisons that rely on counts of tokens.

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第04章《从词数到语义分析》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-4.html) - 结论依据:摘录中关于词面归一化不能合并全部同义表达的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:TF–IDF vectors count the terms according to their exact spelling in a document. So texts that restate the same meaning will have completely different TF–IDF vector representations if they spell things differently or use different words. This messes up search engines and document similarity comparisons that rely on counts of tokens.