robots.txt vs. meta name="ROBOTS" content="ALL, INDEX, FOLLOW
 

I have seen many sites with either one or the other.. some with both. I am curious as to which is better and if the sites with no robot.txt is doing fine why is it needed(other than to prohibit bots from indexing certain areas). And if the site with meta robot text is doin fine why the robot txt?

SE bots look for a file named robots.txt. You can draw your own conclusion from that. :)

In addition, the sites that have both are probably leaving secondary instructions for things to exclude on any particular pages. ;)

This two things can't be compared. Robots.txt file is used to disallow search engine bots to crawl the certain partitions of your website. And meta tag "robots" with content "all, index, follow" is useles, because ALL bots index permissible pages by default.
But if you want to exclude, for example, only one page of your website without robots.txt, you need to change CONTENT to "no index, no follow".

This may sound hugely newb, but which kinds of things wouldn't you want a robot to index?

well what about having neither robot.txt or the meta tag for follow index. it would seem that some have neither as well and are getting along great. just very curious. thanks

Practically every site has folders and files that need to be hidden. For example, administration panel or stuff's login forms.
Here's google's list. He is hidding files from himself.
http://google.com/robots.txt
It means that all the pages of your site will be indexed by search bots.

Your personal home sex video. |jester|