原书出处
Designing Large Language Model Applications
Postprocessing Note that different models will execute different steps within these four components. Normalization Different types of normalization applied include: Converting text to lowercase (if you are using an uncased model) Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)')
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html
EPUB SHA-256:1b4d2347fec2f949fcddfbaf4fad597a1ae20887a096cfae64c59a619b4781bf