原书出处
System Design Interview – An insiders guide, Second Edition
09 网络爬虫
Robots.txt Robots.txt, called Robots Exclusion Protocol, is a standard used by websites to communicate with crawlers. It specifies what pages crawlers are allowed to download. Before attempting to crawl a web site, a crawler should check its corresponding robots.txt first and follow its rules. To avoid repeat downloads of robots.txt file, we cache the results of the file. The file is downloaded and saved to
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_003.html
EPUB SHA-256:ffc3a97753094acb30719f26589af322724b6c17c10371f0626e61687734b345