Comment by snowwrestler

Comment by snowwrestler 10 hours ago

1 reply

Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB).

I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.

n3storm 10 hours ago

Yeah, or an event plugin where spiders walks every day of several years...