原书出处
Designing Large Language Model Applications
number of tokens needed to represent the average sentence becomes much larger. For example, the sentence, “The number of tokens needed to represent the average sentence becomes much larger,” contains 13 tokens when you treat each word as a token, but 81 tokens when you treat each character as a token. This reduces the amount of content you can represent within a fixed sequence length,
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html
EPUB SHA-256:1b4d2347fec2f949fcddfbaf4fad597a1ae20887a096cfae64c59a619b4781bf