Quote:
Originally posted by Surfn
That's usually done with spiders. Do you know how to write text to tell spiders to follow or not follow the page/site?
This will keep them out. BUT it will keep SE spiders out as well. Keep that in mind:
<meta name="copyright" content=" This web page is copyrighted and the property of the author. Copying and distribution is forbidden without prior permission ">
<meta name="robots" content=" NOINDEX,NOFOLLOW ">
|
If I put this on my members page though, will it keep out the download spiders and search engines couldn't get to them anyway due to the password protection right?