知识卡片

普通读书笔记卡 · 6655 · 生成模型:GPT-6

词元切分决定后续可用的信息

理解等级:中级

文档可按段、句、短语和词元等粒度切分,词元具体取字符、子词或词需配合任务。切分不仅是字符串处理,它决定计数、模型输入及输出定位的坐标;词元不自动等于一个完整概念,单个片段也不包含全文语境。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • · 查看原书依据

      In NLP, tokenization is a particular kind of document segmentation. Segmentation breaks up text into smaller chunks or segments. The segments of text have less information than the whole. Documents can be segmented into paragraphs, paragraphs into sentences, sentences into phrases, and phrases into tokens (usually words and punctuation). In this chapter, we focus on segmenting text into tokens with a tokenizer.

    参考来源

    - 位置:《Natural Language Processing in Action, Second Edition》第02章《思维的词元》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-2.html) - 结论依据:摘录中关于词元切分决定后续可用的信息的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。 - 原始内容:In NLP, tokenization is a particular kind of document segmentation. Segmentation breaks up text into smaller chunks or segments. The segments of text have less information than the whole. Documents can be segmented into paragraphs, paragraphs into sentences, sentences into phrases, and phrases into tokens (usually words and punctuation). In this chapter, we focus on segmenting text into tokens with a tokenizer.