返回主题:深网采集与搜索策略

知识卡片

问题卡 · 4457

抓取遵守站点排除规则

理解等级:高级

爬虫在下载前读取 robots.txt,按站点声明筛选允许范围,并缓存及定期更新规则。它是抓取行为约束,不能因为目标链接已进入队列就忽略;书中规则示例属于历史文本,不代表站点当前配置。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • 网络爬虫 · 查看原书依据

      Robots.txt Robots.txt, called Robots Exclusion Protocol, is a standard used by websites to communicate with crawlers. It specifies what pages crawlers are allowed to download. Before attempting to crawl a web site, a crawler should check its corresponding robots.txt first and follow its rules. To avoid repeat downloads of robots.txt file, we cache the results of the file. The file is downloaded and saved to

    参考来源

    - 位置:《System Design Interview – An insiders guide, Second Edition》第09章《网络爬虫》(来源项目 book-sources;源文件:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_003.html) - 结论依据:原文在该位置给出所述对象、机制或比较,支持本卡的解释;原书未展开的实现不作为已证明结论,算式与一致性边界在正文明确限定。 - 原始内容:Robots.txt Robots.txt, called Robots Exclusion Protocol, is a standard used by websites to communicate with crawlers. It specifies what pages crawlers are allowed to download. Before attempting to crawl a web site, a crawler should check its corresponding robots.txt first and follow its rules. To avoid repeat downloads of robots.txt file, we cache the results of the file. The file is downloaded and saved to