知识卡片
训练频率可能影响任务行为
高频任务、输入或预期输出可能更容易被模型处理,低概率但按指令正确的结果则可能被改成更常见表述。需要用保持规则、改变熟悉度的评估区分机械规律与分布熟悉性;相关现象不证明所有错误只由频率导致。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 预训练数据
原书出处 · 1 处
- · 查看原书依据
what assumptions can we make from it about downstream performance? It turns out that there is a correlation between the model’s performance on a given task or input and the pre-training dataset frequency of the task or the salient words in the input, respectively. First observed by Razeghi et al., this phenomenon has been studied in detail in McCoy et al.’s “Embers of Autoregression” paper.
参考来源
- 位置:《Designing Large Language Model Applications》第02章《预训练数据》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch02.html)
- 结论依据:摘录中关于训练频率可能影响任务行为的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:what assumptions can we make from it about downstream performance? It turns out that there is a correlation between the model’s performance on a given task or input and the pre-training dataset frequency of the task or the salient words in the input, respectively. First observed by Razeghi et al., this phenomenon has been studied in detail in McCoy et al.’s “Embers of Autoregression” paper.