原书出处
Natural Language Processing in Action, Second Edition
>>> df['n'] = [len(tok) for tok in vectorizer.vocabulary_] >>> df[df['n'] > 1].sort_values('total').tail() Trust me... There's ... total n ur 8 4 12 2 en 10 3 13 2 an 14 5 19 2 uc 11 9 20 2 e 18 8 26 2 The next round of preprocessing in the BPE tokenizer would retain the character 2-grams en, an, and even e . Then, the
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html
EPUB SHA-256:0e9a35e49ca446e322e5524e53f39fb6bee5b2b23d9cda10ef052aca770b0b36