← 返回卡片

原书出处

Designing Large Language Model Applications

First Edition · 核验版本的摘录快照 · 导入于 2026-11-14T13:54:13.932Z

Along with the pre-tokens (or word tokens), the character offsets are returned. The T5 pre-tokenizer splits only on whitespace, doesn’t collapse multiple spaces into one, and doesn’t split on punctuation or numbers. The behavior can be vastly different for other tokenizers. Tokenization After the optional pre-tokenization step, the actual tokenization step is performed. Some of the important algorithms in this space are byte pair encoding

此预览只包含定位所需的摘录和上下文。

原书定位详情

来源:a19c7933-4eda-4176-aebd-30fe7e1ad292

正文位置:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch03.html

EPUB SHA-256:1b4d2347fec2f949fcddfbaf4fad597a1ae20887a096cfae64c59a619b4781bf