|
robot.txt file
Can someone explain the value of robot.txt. Is it important?
I have created a robot.txt file and uploaded it into the root of my domain but when I do a search for it using one of the many tools available I get an error message:
robots.txt file does not appear to be valid
and the text points to my 404 page. What am I doing wrong? Should I be concerned?
|