|
On my domains I block all known offline browsers, email harvesters, download managers, etc.
Curl is one of those that I block... because I don't want anyone 'mirroring' my content.
I've tried using scripts to clean out the 404s and redirects, but nothing is 100% accurate. Even manual checking isn't perfect, as you could check at the time the server is going thru a reset for whatever reason.
You should use (and trust) whichever you find the most satisfactory for you... or a combination of 2 or 3.
Just my 2c worth.
__________________
Playboy Webmasters - The name says it all! $35 per signup or 60% revshare.
|