Plate,
Many, if not most Search ignore alot of meta tags these days due to tag-spamming...
That being said properly-done meta tags are a vital part of any page.
Robots tag is a valid tag even for google.. If you want a page not spidered or not indexed, then Google bot "should" obey that tag.
I am not Google... but I believe Google wants to spider as many pages as it can.. it then judges the value of those pages before listing them and giving them rank.
Many sites have 1000's of pages in Google, but only a few of the main pages may be well ranked..
eg:
http://www.greenguysboard.com/
Looing at the backlinks to GGandJ -
http://www.google.com/search?sourcei...ndjim%2Ecom%2F
Only 143 are listed of the 160 that link.. eg : Google has deemed some of them to be "not worthy"
DD