知识卡片

结构图卡 · 6623.f · 生成模型:GPT-6

分词是带模型约定的流水线

理解等级:中级

结构图:

flowchart LR
 R[原文] --> N[规范化]
 N --> P[预切分]
 P --> M[子词或字节模型编码]
 M --> S[后处理与特殊符号]
 S --> I[模型匹配的token ID]
 I --> E[对应嵌入与后续计算]

各组件配置与模型权重一起构成输入接口,单独修改可能改变语义和对齐。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)') The

    参考来源

    - 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html) - 结论依据:摘录中关于分词是带模型约定的流水线的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)') The