知识卡片

普通读书笔记卡 · 6605.b · 生成模型:GPT-6

缩放规律依资源约束解释

理解等级:高级

模型参数、训练token与计算量的经验缩放关系,描述特定设置下如何分配预算。固定训练计算的最优分配,不等于固定部署模型大小的最优分配;后者可能用更多训练数据换取较小模型的推理成本优势。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      out that Kaplan et al. underestimated the impact of data size, resulting in language models from that era being significantly undertrained. They showed that to optimize performance of an LLM at a fixed compute budget (called compute-optimal), the training data size needs to increase at the same proportion as the model size. This led to the newer generation of models being trained on more data.

    参考来源

    - 位置:《Designing Large Language Model Applications》第01章《大语言模型入门》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch01.html) - 结论依据:摘录中关于缩放规律依资源约束解释的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:out that Kaplan et al. underestimated the impact of data size, resulting in language models from that era being significantly undertrained. They showed that to optimize performance of an LLM at a fixed compute budget (called compute-optimal), the training data size needs to increase at the same proportion as the model size. This led to the newer generation of models being trained on more data.