It’s a disconcerting feeling, discovering your WordPress site is being relentlessly, almost obsessively, crawled – and not by helpful Googlebot!
Often, it’s not malicious immediately, but a barrage of requests can seriously impact server resources, slowing down everything for genuine visitors. One common culprit? Broken plugins. Seriously, a poorly coded plugin, perhaps one abandoned by its developer years ago, could be unleashing a runaway loop of requests. Think of it like a tiny, digital hamster wheel, spinning endlessly. Then you’ve got bots – and not the good kind.
Plenty of scrapers aim to steal content, and many security scanners, while well-intentioned, can be extremely aggressive. Don’t forget XML-RPC! If it’s enabled (and not properly secured) it’s a prime target for brute-force attacks and spam, resulting in a constant influx of crawl requests. Or… it could be something bizarrely specific to your theme, a deeply nested loop causing infinite requests, or even a misconfigured caching plugin unintentionally serving stale, yet crawlable, content. Checking your server logs is paramount – look for repeated requests from the same IP address, or patterns of requests targeting specific files or pages.
You might also find that a rogue scheduled task is generating all the traffic, a weird scheduling error that went unnoticed. It’s a digital detective game, honestly, and finding the source requires a bit of patient investigation, possibly involving tools like Google Search Console or a plugin specifically designed to monitor bot traffic.