原书出处
System Design Interview – An insiders guide, Second Edition
09 网络爬虫
2. Spider traps A spider trap is a web page that causes a crawler in an infinite loop. For instance, an infinite deep directory structure is listed as follows: www.spidertrapexample.com/foo/bar/foo/bar/foo/bar/… Such spider traps can be avoided by setting a maximal length for URLs. However, no one-size-fits-all solution exists to detect spider traps. Websites containing spider traps are easy to identify due to an unusually large
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/System Design Interview – An insiders guide, Second Edition/_epub-src/text/part0023_split_003.html
EPUB SHA-256:ffc3a97753094acb30719f26589af322724b6c17c10371f0626e61687734b345