知识卡片

结构图卡 · 6661.e · 生成模型:GPT-6

词元表示沿两条方向压缩

理解等级:中级

结构图:

flowchart LR
 A[原始文本] --> B[分词与可选归一化]
 B --> C[带词表坐标的词元序列]
 C --> D[位置编号或独热序列]
 C --> E[按词表聚合]
 E --> F[二元出现向量]
 E --> G[词项计数向量]
 D --> H[保留词元顺序]
 F --> I[丢词序保留出现]
 G --> J[丢词序保留次数]

序列编码是否能还原原文,取决于切分前是否已丢失信息;密集独热不是唯一序列表示。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      One nice feature of this vector representation of words and tabular representation of documents is that no information is lost. The exact sequence of tokens is encoded in the order of the one-hot vectors in the table representing a document. As long as you keep track of which words are indicated by which column, you can reconstruct the original sequence of tokens from this table

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html) - 结论依据:摘录中关于词元表示沿两条方向压缩的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:One nice feature of this vector representation of words and tabular representation of documents is that no information is lost. The exact sequence of tokens is encoded in the order of the one-hot vectors in the table representing a document. As long as you keep track of which words are indicated by which column, you can reconstruct the original sequence of tokens from this table