Link Checkers, Site stealers and User_Agents
 

I use htaccess protection to guard against the constant stream of site stealer agents sent to my domains and the htaccess file blocks by user agent.
I have included most 'known' stealer software agents in the file.

It seems we have a problem with some of the user_agents being employed by some link lists to check links.
Recently one or two linklists have been dropping some of our links and when we enquire about it we discover that the user_agent employed is one of those which is commonly used by site stealers.

I have two suggestions which would save a lot of time, trouble and mistaken removal of valid links in a case like this:
1. Change the user agent name (name it after your linklist).
2. Display the name of your user_agent on your submissions page so that webmasters like us can decide whether it's worthwhile allowing the agent.

It seems obvious to me that renaming the user agent would be the best solution but I have no idea how hard it is to do (I know it's possible because at least one link lists did this in order to overcome the problem) but - for some, maybe option 2 is the only answer...

Perhaps there's another way to guard against site stealers?
If not - does anybody have any better suggestions?

No way.

The last thing that I would want is someone to know what to look for so that they can serve up the correct page to me and only me.

This is the same reason that I don't want anyone to know what IP address that I use so I never posts on boards that show IPs.

That's fair enough, but what about option 1?

...or

What else do you suggest?

Actually we've never had a problem with Cleos links.

I sure as hell wouldn't go to the trouble of cloaking a free site for one link list - that would be anal.

It still leaves us with the same problem for the "others" though...
Altering the user_agent slightly will allow it to bypass the htaccess protection - no need to say which user_agent you use but if you use a link checker script which is in the common list of site stealer bots then you will not get access!