知识卡片

问题卡 · 4452

爬取流水线分离职责

理解等级:中级

调度队列提供待抓 URL,下载器处理网络请求,解析器验证和抽取内容,过滤与去重决定哪些链接重新入队。将这些步骤分开便于扩容和处理畸形页面,不把慢解析直接阻塞所有下载工作。

这张卡片看不懂?

已有反馈 · 0 / 10 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • 网络爬虫 · 查看原书依据

      Web crawler workflow To better explain the workflow step-by-step, sequence numbers are added in the design diagram as shown in Figure 9-4. Step 1: Add seed URLs to the URL Frontier Step 2: HTML Downloader fetches a list of URLs from URL Frontier. Step 3: HTML Downloader gets IP addresses of URLs from DNS resolver and starts downloading. Step 4: Content Parser parses HTML pages

    参考来源

    - 位置:《System Design Interview – An insiders guide, Second Edition》第09章《网络爬虫》(来源项目 book-sources;源文件:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_002.html) - 结论依据:原文在该位置给出所述对象、机制或比较,支持本卡的解释;原书未展开的实现不作为已证明结论,算式与一致性边界在正文明确限定。 - 原始内容:Web crawler workflow To better explain the workflow step-by-step, sequence numbers are added in the design diagram as shown in Figure 9-4. Step 1: Add seed URLs to the URL Frontier Step 2: HTML Downloader fetches a list of URLs from URL Frontier. Step 3: HTML Downloader gets IP addresses of URLs from DNS resolver and starts downloading. Step 4: Content Parser parses HTML pages