Posted by Steve Nyhof on 12 December 2009 | 101 Comments
This will be quick and will be expanded on as I move forward. Our CMS does not include a robots.txt file. You are more than welcome to add and upload one to your account. However, this system makes use of the google sitemap.xml file which tells Google Bots what to index or not.
Some people would be concerned about the image folders being accessed and indexed in Google images. If you know how to take advantage of this you will begin today giving your images a more descriptive name and let the bots have their way.
Example...
- image001.jpg does very little to tell anyone what the image is.
- landingpages-logo-steve-nyhof.jpg does a great deal to tell what the image is and who it belongs to.
I strongly recommend that you go to your image folder inside the CMS and rename the images accordingly. The system will automatically update the links on your pages with the new name.
So to conclude, we do not use a robots.txt file by default. But you may if you like.
See Google Webmaster Guidelines for more information