Link Checkers, Site stealers and User_Agents
I use htaccess protection to guard against the constant stream of site stealer agents sent to my domains and the htaccess file blocks by user agent.
I have included most 'known' stealer software agents in the file.
It seems we have a problem with some of the user_agents being employed by some link lists to check links.
Recently one or two linklists have been dropping some of our links and when we enquire about it we discover that the user_agent employed is one of those which is commonly used by site stealers.
I have two suggestions which would save a lot of time, trouble and mistaken removal of valid links in a case like this:
1. Change the user agent name (name it after your linklist).
2. Display the name of your user_agent on your submissions page so that webmasters like us can decide whether it's worthwhile allowing the agent.
It seems obvious to me that renaming the user agent would be the best solution but I have no idea how hard it is to do (I know it's possible because at least one link lists did this in order to overcome the problem) but - for some, maybe option 2 is the only answer...
Perhaps there's another way to guard against site stealers?
If not - does anybody have any better suggestions?
|