知识卡片

问题卡 · 4451

URL与内容分别去重

理解等级:高级

URL 去重避免同一地址重复入队和形成循环,内容去重避免不同地址的同一页面被反复存储。两者处理对象不同:只比较地址不能消除镜像内容,只比较内容也不能阻止重复下载。

这张卡片看不懂?

已有反馈 · 0 / 10 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • 网络爬虫 · 查看原书依据

      URL Seen? “URL Seen?” is a data structure that keeps track of URLs that are visited before or already in the Frontier. “URL Seen?” helps to avoid adding the same URL multiple times as this can increase server load and cause potential infinite loops. Bloom filter and hash table are common techniques to implement the “URL Seen?” component. We will not cover the detailed implementation

    参考来源

    - 位置:《System Design Interview – An insiders guide, Second Edition》第09章《网络爬虫》(来源项目 book-sources;源文件:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_002.html) - 结论依据:原文在该位置给出所述对象、机制或比较,支持本卡的解释;原书未展开的实现不作为已证明结论,算式与一致性边界在正文明确限定。 - 原始内容:URL Seen? “URL Seen?” is a data structure that keeps track of URLs that are visited before or already in the Frontier. “URL Seen?” helps to avoid adding the same URL multiple times as this can increase server load and cause potential infinite loops. Bloom filter and hash table are common techniques to implement the “URL Seen?” component. We will not cover the detailed implementation