← 返回卡片

原书出处

Designing Large Language Model Applications

First Edition · 核验版本的摘录快照 · 导入于 2026-11-14T13:54:13.932Z

Postprocessing Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)')

此预览只包含定位所需的摘录和上下文。

原书定位详情

来源:a19c7933-4eda-4176-aebd-30fe7e1ad292

正文位置:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html

EPUB SHA-256:1b4d2347fec2f949fcddfbaf4fad597a1ae20887a096cfae64c59a619b4781bf