Botnets keep sending DuckDuckGo weirder and weirder traffic, and frankly I don't get it. For a while now I've seen a lot of requests like these:
I suppose those forms make some sense. I presume they are looking for sites running exploitable software, and so they set up automated queries to search engines to find new sites.
However, what doesn't make sense is sending the same query hundreds of thousands of times a day from each machine. Someone presumably took the time to carefully construct these queries, given that they generally appear to be in the right form. And yet they send back the same results a tenth of a second later, so why would you keep repeating them? A computer will pop-up and will just start hammering on the query. If I unblock it even days later, it is still doing it.
But that's not the weirdest behavior. For the past several weeks I've been getting tons of the exact same request:
These requests come in slower per machine but from a much greater number of machines. I honestly don't understand the point of them at all. Does anyone out there?with the referrer: http://www.elahmad.com/tv/Webtv/Arabian.htm
As you may know, DuckDuckGo does not save IPs (here's how). So if you're wondering how we go about blocking them, it happens all at the firewall level, which is dissociated from query data. If we didn't block the most egregious botnet machines and abusers, our machines would almost instantly be under water.
This discussion now makes me wonder if other search engines include this errant traffic in their query counts. We work hard to keep them completely out because they would overwhelm our real direct queries #s and therefore distort our perception of progress. We also separate out API requests for the same reason, which now also makes me wonder whether everyone else is doing that too.