Quote:
|
This may sound hugely newb, but which kinds of things wouldn't you want a robot to index?
|
Practically every site has folders and files that need to be hidden. For example, administration panel or stuff's login forms.
Here's google's list. He is hidding files from himself.
http://google.com/robots.txt
Quote:
|
well what about having neither robot.txt or the meta tag for follow index. it would seem that some have neither as well and are getting along great. just very curious. thanks
|
It means that all the pages of your site will be indexed by search bots.