Comments on: Google Webmaster Tools And Robots.txt http://www.g34media.com/2008/10/18/google-webmaster-tools/ New Media Services Thu, 29 Jul 2010 13:17:11 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Blog for Beginners http://www.g34media.com/2008/10/18/google-webmaster-tools/#comment-1344 Blog for Beginners Sat, 25 Oct 2008 02:57:22 +0000 http://www.g34media.com/?p=124#comment-1344 Hey Missy I agree the article by JR on Google'WMT is awesome. On last check, you don't have a robots.txt file yet which is not an ideal situation for you. I guess JR has explained to you what it's for.... The other 19 urls could be some of the broken links within your site that you need to fix. Use a broken link checker plugin to see where the link is and rectify it immediately. Hope it helps... Yan <abbr><em>Blog for Beginners’s last blog post..<a href="http://thoushallblog.com/carl-ocab-direct-response-templates/" rel="nofollow">Me, Carl Ocab and Direct Response Templates</a></abbr></em> Hey Missy

I agree the article by JR on Google’WMT is awesome. On last check, you don’t have a robots.txt file yet which is not an ideal situation for you. I guess JR has explained to you what it’s for….

The other 19 urls could be some of the broken links within your site that you need to fix. Use a broken link checker plugin to see where the link is and rectify it immediately.

Hope it helps…

Yan

Blog for Beginners’s last blog post..Me, Carl Ocab and Direct Response Templates

]]> By: Missy http://www.g34media.com/2008/10/18/google-webmaster-tools/#comment-1335 Missy Mon, 20 Oct 2008 15:47:38 +0000 http://www.g34media.com/?p=124#comment-1335 Hey, thanxs JR. I think i better understand now what is the robots.txt file. (or atleast its purpose) You say it needs to be changed via my host, which file is it in? Thanxs for the copy of your robots file, will look it over. That is awesomesauce! As for the 19 URL's not found, i will have to look into it. I honestly don't know. Thanxs for your groovy help. Appreciate it loads. Hey, thanxs JR. I think i better understand now what is the robots.txt file. (or atleast its purpose)

You say it needs to be changed via my host, which file is it in?

Thanxs for the copy of your robots file, will look it over. That is awesomesauce!

As for the 19 URL’s not found, i will have to look into it. I honestly don’t know.

Thanxs for your groovy help. Appreciate it loads.

]]> By: JR @ Internet Marketing Strategies http://www.g34media.com/2008/10/18/google-webmaster-tools/#comment-1332 JR @ Internet Marketing Strategies Sun, 19 Oct 2008 07:22:27 +0000 http://www.g34media.com/?p=124#comment-1332 Hello Missy, Thanks for the mention, I appreciate it. RE: The URL's blocked on your robots file just look at your robots file and see what pages are blocked. You can see what your robots file says in GWT, but you can only edit it from your hosting files. Usually a robots.txt file is created by the webmaster, so if you did not create it, I would certainly look into what is in yours and edit it correctly. There are pages that you do not want indexed and that you do not want Google to crawl to preserve link juice and crawl juice for the important pages. You do not want for instance, your css files, wp-admin, feeds, about, contact pages etc...to be crawled and indexed. You need to block categories and tags on blogs because these would be considered duplicate content, all this is accomplished with a robots file. Here is my robots file, you can copy it and use that for your blog, edit out pages you may not have as needed, such as terms-of-service, that you may not have on your blog. User-agent: * Disallow: /wp-admin/ Disallow: /cgi-bin/ Disallow: /*.css$ Disallow: /cont/ Disallow: /scripts/ Disallow: /wp-content/themes/ Disallow: /?s= Disallow: /rss/ Disallow: /feed/ Disallow: /feed/ Disallow: /terms of use/ Disallow: /disclaimer/ Disallow: /privacy policy/ Disallow: /search/ Disallow: /about/ Disallow: /contact me/ Disallow: /tags/ Disallow: /archives/ Disallow: /category/ Disallow: /trackback/ Disallow: /contact-form/ Disallow: /comments/feed/ Disallow: /*.avi$ Disallow: /*.cgi$ Disallow: /index.php Disallow: /*.js$ Disallow: /*.inc$ User-agent: Mediapartners-Google* The 19 not found URL's may be redirects, what are they? and What is the reason that GWT is showing them not found is a good place to start. Hello Missy,
Thanks for the mention, I appreciate it. RE: The URL’s blocked on your robots file just look at your robots file and see what pages are blocked.

You can see what your robots file says in GWT, but you can only edit it from your hosting files. Usually a robots.txt file is created by the webmaster, so if you did not create it, I would certainly look into what is in yours and edit it correctly.

There are pages that you do not want indexed and that you do not want Google to crawl to preserve link juice and crawl juice for the important pages. You do not want for instance, your css files, wp-admin, feeds, about, contact pages etc…to be crawled and indexed. You need to block categories and tags on blogs because these would be considered duplicate content, all this is accomplished with a robots file.

Here is my robots file, you can copy it and use that for your blog, edit out pages you may not have as needed, such as terms-of-service, that you may not have on your blog.
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /*.css$
Disallow: /cont/
Disallow: /scripts/
Disallow: /wp-content/themes/
Disallow: /?s=
Disallow: /rss/
Disallow: /feed/
Disallow: /feed/
Disallow: /terms of use/
Disallow: /disclaimer/
Disallow: /privacy policy/
Disallow: /search/
Disallow: /about/
Disallow: /contact me/
Disallow: /tags/
Disallow: /archives/
Disallow: /category/
Disallow: /trackback/
Disallow: /contact-form/
Disallow: /comments/feed/
Disallow: /*.avi$
Disallow: /*.cgi$
Disallow: /index.php
Disallow: /*.js$
Disallow: /*.inc$

User-agent: Mediapartners-Google*

The 19 not found URL’s may be redirects, what are they? and What is the reason that GWT is showing them not found is a good place to start.

]]>