知识卡片
元数据边界保留章节作用域
标题、段落和小节可指导切块,并作为元数据或前缀传给下游。必须保留正确层级和作用范围,不能把页眉当新章节,也不能因相邻文本被合并就丢掉来源定位。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
- · 表示学习与嵌入
原书出处 · 1 处
- · 查看原书依据
Metadata-Aware Chunking Any metadata that you have about the document can be leveraged to determine chunking boundaries. Useful metadata information includes paragraph boundaries, section and subsection boundaries, etc. If the metadata isn’t already available, you might need to use document parsing techniques to extract this information. Several libraries can facilitate this, including Unstructured. Layout-Aware Chunking A more involved form of metadata-aware chunking is layout-aware chunking.
参考来源
- 位置:《Designing Large Language Model Applications》第11章《表示学习与嵌入》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Designing Large Language Model Applications/_epub-src/OEBPS/ch11.html)
- 结论依据:摘录中关于元数据边界保留章节作用域的定义、比较或条件支撑卡片;原书表述不严谨的范围已在覆盖记录逐项限定,不把案例结果外推为普遍结论。
- 原始内容:Metadata-Aware Chunking Any metadata that you have about the document can be leveraged to determine chunking boundaries. Useful metadata information includes paragraph boundaries, section and subsection boundaries, etc. If the metadata isn’t already available, you might need to use document parsing techniques to extract this information. Several libraries can facilitate this, including Unstructured. Layout-Aware Chunking A more involved form of metadata-aware chunking is layout-aware chunking.