知识卡片
词元和语法是处理文本的不同层
词元是后续处理采用的基本文本单元,语法规定这些单元如何组成结构。编程语言和自然语言都可作这种分析,但词元不只包括保留关键字,自然语言的词元也不必与日常词一一对应;分词策略需随语言、任务与模型约定。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- · 查看原书依据
Programming languages are very similar to natural languages, like English. Both kinds of languages are used to communicate instructions from one information processing system to another. Both can communicate thoughts from human to human, human to machine, or even machine to machine. Both natural and programming languages feature the concept of tokens, which you can think of as words, for now. Whether your text is
参考来源
- 位置:《Natural Language Processing in Action, Second Edition》第01章《会读写的机器与自然语言处理》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Natural Language Processing in Action, Second Edition/_epub-src/OEBPS/Text/chapter-1.html)
- 结论依据:摘录中关于词元和语法是处理文本的不同层的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Programming languages are very similar to natural languages, like English. Both kinds of languages are used to communicate instructions from one information processing system to another. Both can communicate thoughts from human to human, human to machine, or even machine to machine. Both natural and programming languages feature the concept of tokens, which you can think of as words, for now. Whether your text is