Link Bots

[cd34]

You can have it spider a page where you have access to the raw logs. Then grep the weblogs for the page that you had it spider.

You can:
grep -r User-Agent: *

in the directory where the script resides.

You could run ettercap, tcpdump or some other packet capture program on the server to watch packets as it does a check. Might be a bit overwhelming as a LOT of stuff might be going on at the same time. Should require root access to do this -- only for the insanely curious.

There are a bunch of other ways -- a cgi script that dumps the environment to a location on the server, or emails you the info when someone hits it, etc.