知识卡片

问题卡 · 4450

爬虫目标决定规模边界

理解等级:高级

爬虫从种子页面提取链接,再调度下载更多页面。搜索索引、归档和监控对内容类型、更新频率、留存与去重有不同需求;应先约定目标,不能把一套遍历流程当成所有爬取业务的完整方案。

这张卡片看不懂?

已有反馈 · 0 / 10 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • 网络爬虫 · 查看原书依据

      Step 1 - Understand the problem and establish design scope The basic algorithm of a web crawler is simple: 1. Given a set of URLs, download all the web pages addressed by the URLs. 2. Extract URLs from these web pages 3. Add new URLs to the list of URLs to be downloaded. Repeat these 3 steps. Does a web crawler work truly as simple

    参考来源

    - 位置:《System Design Interview – An insiders guide, Second Edition》第09章《网络爬虫》(来源项目 book-sources;源文件:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_001.html) - 结论依据:原文在该位置给出所述对象、机制或比较,支持本卡的解释;原书未展开的实现不作为已证明结论,算式与一致性边界在正文明确限定。 - 原始内容:Step 1 - Understand the problem and establish design scope The basic algorithm of a web crawler is simple: 1. Given a set of URLs, download all the web pages addressed by the URLs. 2. Extract URLs from these web pages 3. Add new URLs to the list of URLs to be downloaded. Repeat these 3 steps. Does a web crawler work truly as simple