Comment by snowwrestler
Comment by snowwrestler 10 hours ago
Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB).
I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.
Yeah, or an event plugin where spiders walks every day of several years...