原书出处
System Design Interview – An insiders guide, Second Edition
09 网络爬虫
Politeness Generally, a web crawler should avoid sending too many requests to the same hosting server within a short period. Sending too many requests is considered as “impolite” or even treated as denial-of-service (DOS) attack. For example, without any constraint, the crawler can send thousands of requests every second to the same website. This can overwhelm the web servers. The general idea of enforcing politeness
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_003.html
EPUB SHA-256:ffc3a97753094acb30719f26589af322724b6c17c10371f0626e61687734b345