← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 6623.a · 生成模型:GPT-6

拼写变体会改变序列形状

理解等级:中级

一个拼写错误可能被拆成更多子词,使序列更长、词元频率更低。模型有时可凭上下文恢复意思,但不能保证;诊断领域文本失败时可以比较正常写法与变体的切分,仍需独立确认错误原因。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      brittle with typos. But at least the OOV problem has been dealt with by breaking the words into subwords. The vocabulary also doesn’t seem to have an entry for the word “corduroy,” thus confirming its poor sense of fashion. Meanwhile, note that there is a distinct token for three contiguous exclamation points, which is different from the token that represents a single exclamation point. Semantically,

    参考来源

    - 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html) - 结论依据:摘录中关于拼写变体会改变序列形状的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:brittle with typos. But at least the OOV problem has been dealt with by breaking the words into subwords. The vocabulary also doesn’t seem to have an entry for the word “corduroy,” thus confirming its poor sense of fashion. Meanwhile, note that there is a distinct token for three contiguous exclamation points, which is different from the token that represents a single exclamation point. Semantically,