知识卡片
初始化须保留预训练参数
Xavier可初始化某些矩阵权重,是从零训练的设置之一。对整个模型apply会遍历子模块,若已加载预训练权重后再调用,会破坏复用起点;初始化对象、时点与参数类型应明确记录。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- · 查看原书依据
Also, define an init_weights() method that can be called to initialize the weights of all submodules of the transformer, as shown in listing 9.12. Xavier initialization is commonly used for transformers, so use it here. The PyTorch nn.Module documentation23 describes the apply(fn) method that recursively applies fn to every submodule of the caller.
参考来源
- 位置:《Natural Language Processing in Action, Second Edition》第09章《可堆叠的Transformer》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-9.html)
- 结论依据:摘录中关于初始化须保留预训练参数的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Also, define an init_weights() method that can be called to initialize the weights of all submodules of the transformer, as shown in listing 9.12. Xavier initialization is commonly used for transformers, so use it here. The PyTorch nn.Module documentation23 describes the apply(fn) method that recursively applies fn to every submodule of the caller.