原书出处
Designing Large Language Model Applications
Stripping off accents from characters, like from the word Peña Unicode normalization Let’s see what kind of normalization is applied on the uncased version of BERT: tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") print(tokenizer.backend_tokenizer.normalizer.normalize_str( 'Pédrò pôntificated at üs:-)') The output is: pedro pontificated at us:-) As we can see, the accents have been removed and the text has been converted to lowercase. There isn’t much normalization done in tokenizers
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html
EPUB SHA-256:1b4d2347fec2f949fcddfbaf4fad597a1ae20887a096cfae64c59a619b4781bf