URL details: state.smerity.com/smerity/state/01EAN3YGGXN93GFRM8XWQA9FTR
URL title:
Smerity.com: Crawling in three stories
URL description:
Inspired by A Facebook crawler was making 7M requests per day to my stupid website (Hacker News), I thought it worth revisiting my history with web crawlers. This will take place from a few specific stories. For multiple stories the names of the companies have been removed.
URL last crawled:
2022-06-29
URL speed:
0.460 MB/s,
downloaded in 0.030 seconds
We found no external links pointing to this url.