知识卡片

普通读书笔记卡 · 6614.c · 生成模型:GPT-6

困惑度筛选要防两端失真

理解等级:中级

高困惑度可能是乱码或域外有效内容,低困惑度也可能是重复模板。只保留最低分会收窄分布;按中间区间采样是一种避免极端的策略,但最佳区间仍需以目标任务和语料诊断验证。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Short, repetitive text can have low perplexity. Note that writing style gets factored into perplexity. If the reference language model is trained over Wikipedia, then documents written in an informal style may receive higher perplexity scores. Therefore, it would be beneficial to have a more involved filtering strategy.

    参考来源

    - 位置:《Designing Large Language Model Applications》第02章《预训练数据》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch02.html) - 结论依据:摘录中关于困惑度筛选要防两端失真的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Short, repetitive text can have low perplexity. Note that writing style gets factored into perplexity. If the reference language model is trained over Wikipedia, then documents written in an informal style may receive higher perplexity scores. Therefore, it would be beneficial to have a more involved filtering strategy.