Fast 404 error checker for Link Lists

[SortLinks]

oast, how do you manage to dist. good bots from bad ones?

[oast]

Thru the User Agent string that (nearly) all programs use to identify themselves.

I use htaccess then to forbid (or redirect) the 'bad boys'.

[oast]

A small extract from the filtering lines of my .htaccess file looks like this:

RewriteEngine On

# Blank (or "-") Referer *and* User Agent
RewriteCond %{HTTP_REFERER} ^-?$
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule .* - [F]
# Address harvesters
RewriteCond %{HTTP_USER_AGENT} ^(autoemailspider|ExtractorPro) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^E?Mail.?(Collect|Harvest|Magnet|Reaper|Siphon|Sweeper|Wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (DTS.?Agent|Email.?Extrac) [NC]
RewriteRule .* - [F]

When 'code red' was more prevalant than it is today, I redirected all requests to Micro$oft

mod_rewrite can be a very powerful tool if used correctly.

AFAIK Apache is the only server it is available on, but as a large number of hosting companies prefer Apache, you should be OK

[oast]

Originally Posted by oast

When 'code red' was more prevalant than it is today, I redirected all requests to Micro$oft

Should I have said that?

Honestly Bill, I was naive at the time. I don't do things like that any more |blowkiss|