知识卡片

普通读书笔记卡 · 8698.a · 生成模型:GPT-6

池化方式必须匹配模型训练

理解等级:高级

将token表示变成句向量可用CLS、均值、最大、加权均值或末token。应读取具体模型配置与训练目标;本章TAS-B模型确实配置CLS,但这不能推出所有Sentence Transformers模型都用CLS,也不能仅凭BERT有NSP断言句向量一定优于RoBERTa。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      In this example, the embedding is drawn from the [CLS] token of the last layer of the DistilBERT model. Other ways of extracting embeddings from models include: Mean pooling, where the average is taken across all token outputs in the sequence Max pooling, where the maximum value in each dimension across all tokens is taken Weighted mean, where more weight is given to the last

    参考来源

    - 位置:《Designing Large Language Model Applications》第11章《表示学习与嵌入》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch11.html) - 结论依据:摘录中关于池化方式必须匹配模型训练的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:In this example, the embedding is drawn from the [CLS] token of the last layer of the DistilBERT model. Other ways of extracting embeddings from models include: Mean pooling, where the average is taken across all token outputs in the sequence Max pooling, where the maximum value in each dimension across all tokens is taken Weighted mean, where more weight is given to the last