知识卡片
后处理添加模型协议符号
后处理可添加分类、分隔或序列边界等特殊符号,连接文本表示与模型协议。它们应与训练约定和任务模板一致;看起来相同的尖括号字符串可能只是普通文本,不一定是一个受协议识别的特殊ID。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 词表与分词
原书出处 · 1 处
- · 查看原书依据
Postprocessing Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)')
参考来源
- 位置:《Designing Large Language Model Applications》第03章《词表与分词》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html)
- 结论依据:摘录中关于后处理添加模型协议符号的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Postprocessing Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)')