← 返回刚才阅读的位置

知识卡片

问题卡 · 4455

更新历史决定重抓频率

理解等级:中级

网页会持续增删改,爬虫需要重新访问。按更新历史和重要性分配不同重抓频率,可比全站等频重抓更节省资源;需要新鲜度的页面应优先获得预算,而不是只追求一次抓到最多地址。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • 网络爬虫 · 查看原书依据

      Freshness Web pages are constantly being added, deleted, and edited. A web crawler must periodically recrawl downloaded pages to keep our data set fresh. Recrawl all the URLs is time-consuming and resource intensive. Few strategies to optimize freshness are listed as follows: •Recrawl based on web pages’ update history. •Prioritize URLs and recrawl important pages first and more frequently. Storage for URL Frontier In real-world

    参考来源

    - 位置:《System Design Interview – An insiders guide, Second Edition》第09章《网络爬虫》(来源项目 book-sources;源文件:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_003.html) - 结论依据:原文在该位置给出所述对象、机制或比较,支持本卡的解释;原书未展开的实现不作为已证明结论,算式与一致性边界在正文明确限定。 - 原始内容:Freshness Web pages are constantly being added, deleted, and edited. A web crawler must periodically recrawl downloaded pages to keep our data set fresh. Recrawl all the URLs is time-consuming and resource intensive. Few strategies to optimize freshness are listed as follows: •Recrawl based on web pages’ update history. •Prioritize URLs and recrawl important pages first and more frequently. Storage for URL Frontier In real-world