Another way that that the robots.txt file helps you search engine optimize your Drupal site is by allowing you to specify where your sitemaps are located. While you probably want to submit your sitemap directly to Google, Yahoo!, and MSN, it's a good idea to put a reference to it in the robots.txt file for all of those other search engines. You can do this by carrying out the following steps:
1. Open the robots.txt file for editing.
2. The sitemap directive is independent of the User-agent line, so it doesn't matter where you place it in your robots.txt file.
To keep things neat, add this line first:
# Sitemaps
Add these lines for your XML sitemap:
Sitemap: http://www.yourDrupalsite.com/sitemap.xml Sitemap: http://www.yourDrupalsite.com/?q=sitemap.xml
If you're using the URL list sitemap instead, add these lines:
Sitemap: http://www.yourDrupalsite.com/urllist.txt Sitemap: http://www.yourDrupalsite.com/?q=urllist.txt
3. Your finished
4. Save your robots.txt file, uploading it if necessary, replacing the existing file (you backed it up, didn't you?).
5. Go to http://www.yourDrupalsite.com/robots.txt and double-check that your changes are in effect. You may need to perform a refresh on your browser to see the changes.
If you have an XML sitemap, use it. If not, use the URL list sitemap. However, do not add both, an XML sitemap and a URL list sitemap, to the robots.txt file. It could confuse the search engines; possibly even causing duplicate content on your site. Also, do not add your visitor-facing sitemap to your robots.txt file.
Using Google's Webmaster Tools to evaluate your robots.txt file
Warning! The robots.txt file is easy to mess up! It's not written for humans so it's easy for site owners and webmasters to misunderstand exactly how to use it. Take care not to break your SEO campaign simply because a poorly written robots. txt file is excluding your site from Google. Fortunately, Google's Webmaster Tools provides a helpful utility that shows you exactly which pages are being excluded and included by your robots.txt file. Carry out the following steps to evaluate your robots.txt file using Google's Webmaster Tools:
1. Go to http://www.google.com/webmasters/, log in, and click on your site.
2. Click on the Tools menu item and you'll see a screen similar to the following screenshot:

3. Click on Analyze robots.txt. You'll see some interesting statistics about your robots.txt file.
You'll see the following options on the next page:
4. Further down the page, you'll see the text of the robot.txt file that Google last downloaded from your site. If you've tweaked it more recently than the last download, you can copy and paste your changes into the box provided so that you can test your changes. This is for testing purposes only. Any changes you make will not be saved.
5. The next box, labeled Test URLs against this robots.txt file, is a list of URLs from your web site. By making changes to the robots.txt box and adding URLs, you can see how different rules will affect the way Google sees your site.
6. Further down, Choose User-agents allows you to specify which Googlebot you want to evaluate. Google has several they use, like Googlebot-Mobile and Googlebot-Image.
Let's try an example. We're going to tell Googlebot-Image to leave our site alone! As you can see below, I added these lines to the robots.txt text box:
User-agent: Googlebot-Image Disallow: /*.jpg$ Disallow: /*.gif$ Disallow: /*.png$
Both Googlebot and Googlebot-Image were blocked by our robots.txt file. For more information about the robots.txt specification, please visit these sites:
Feel free to try different things on your own. You can't hurt your site here. If you like what you've done, be sure to copy and paste the changes into the robots.txt file on the root level of your Drupal site.
Our website is not responsible for the information contained by this article. Webworldarticles.com is a free articles resource thus practically any visitor can submit an article. However if you notice any copyrighted material, please contact us and we will remove the article(s) in discussion right away.
This article was sent to us by:
Latoya K. at
01172010
1. Robots txt file for Drupal examples and editing
All articles in this directory are property of their respective authors. Additionally, read our Privacy Policy
© 2010 WebWorldarticles.com - All Rights Reserved.