← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6659 · 生成模型:GPT-6

大小写归一化需配合实体与词形

理解等级:中级

合并House和house可减少部分词形差异,也可能混同专名和普通词。只降句首也仍会伤及句首姓名;需根据检索或实体任务选择并保留原文。词表大小和训练样本需求不必恰好翻倍,更无每维必须一条标注的统一法则。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Undoing this denormalization is called case normalization or, more commonly, case folding. Normalizing word and character capitalization is one way to reduce your vocabulary size and generalize your NLP pipeline. It helps you consolidate words that are intended to mean (and be spelled) the same under a single token. However, some information is often communicated by capitalization of a word—for example, doctor and Doctor often

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html) - 结论依据:摘录中关于大小写归一化需配合实体与词形的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Undoing this denormalization is called case normalization or, more commonly, case folding. Normalizing word and character capitalization is one way to reduce your vocabulary size and generalize your NLP pipeline. It helps you consolidate words that are intended to mean (and be spelled) the same under a single token. However, some information is often communicated by capitalization of a word—for example, doctor and Doctor often