New Media Services
Google Webmaster Tools And Robots.txt
Do you use Google Webmaster Tools? The free service from Google to help webmasters make their sites more Google friendly.
I have been thinking of Google Webmaster Tools lately as JR from Internet Marketing Strategies) has written a fabulous article covering the tool. He describes the 6 tabs on the tool one by one and covers some tips on why and how to use GWT.
As i mentioned to JR every time i read a new article on GWT, i learn something new. I wouldn’t necessarily describe GWT as super advanced, but it isn’t exactly child’s play either. Some level of webmastering is required.
I want to get to the bottom of one thing within GWT: robot.txt
The what, who, when, why, and how? Feel free to share your tips and experiences below. When i go to Google Webmaster Tools, i notice a few things in the overview section. (see screenshot below)
It seems this blog has 19 not found URL’s and 2 URL’s restricted by robots.txt
What does this mean? How do i fix it? Feel free to weigh in below in comments.
Missy.
| Print article | This entry was posted by Missy on October 18, 2008 at 8:10 PM, and is filed under Blogging Tips. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site. |
No trackbacks yet.
The Best Sites To Get Free Facebook Buttons, Icons Or Badges
about 6 months ago - 5 comments
You have an awesome Facebook Page, right? And you want the world to know, so they can fan you up. Great! Let’s take a look at a few resources where you can score some free and nice looking Facebook Buttons, Icons or Badges. 1. Social Media Buttons – this site is dedicated to nothing but
SEO For WordPress Bloggers
about 9 months ago - 15 comments
Below are my top five SEO tips for WordPress bloggers. 1. Install an SEO plugin on your blog – such as the powerful All-in-One Seo or the newer more robust Platinum SEO. These two take care of alot of the basic onsite search engine optimization you need on your blog posts and pages. (this mostly
Sell Products Easily On Your WordPress Blog With E-commerce Plugin
about 1 year ago - 2 comments
Want to sell ebooks, wordpress themes, wordpress plugins or other digital content on your blog? If so. Have a look at the fantastic new video from Hungry Dog Media and WordPress.TV, in which a step-by-step is provided on how to set up shop on your blog in under 5 minutes. Yes five minutes! It’s quite
7 Elements Of A Successful Website
about 1 year ago - 16 comments
1. Name – Every website needs a domain name that’s descriptive and relevant, but what type of name should you pick? There are two rules of thought here. One is to go the brand route and use a name that over time will become relevant. Second is to go the route of a keyword-rich name
Have You Heard About The Twitter Remote Widget?
about 1 year ago - 9 comments
I went to Twitter Counter earlier today to grab the code to add their widget on this blog, and found a new addition called Twitter Remote. This new service reminds me of the My Blog Log and Blog Catalog widgets that capture and display recent visitors. Essentially this widget works the same way. Grab the
5 Affiliate Marketing Tips For Bloggersfiv
about 1 year ago - 15 comments
There is loads of information on the web on the topic of affiliate marketing, but if you’re a blogger, the five tips below are all you need to know to get started. If you’re a blogger and want to know how to best use affiliate marketing on your blog, read what i’ve written below, then
Free Gmail Stickers From Google
about 1 year ago - 3 comments
Google is releasing free Gmail stickers to bloggers, well really to anyone who wants them. They announced the cool freebie yesterday on the Gmail blog. Grab some before they are all gone. To request your free set, send a postage paid envelope to: Googleplex P O Box 391420 Mountain View, CA 94039-1420
5 Twitter Tools You Should Be Using If You’re A Blogger
about 1 year ago - 22 comments
Of all the social networks that exist, my fave by far is Twitter. Why? Well because i don’t have to spend my precious time adding dozens of glitter buttons (not that there is anything wrong with that) or filling out numerous quizzes or downloading a million gazillion photos. I just update and go. Love it.
Problogger Launches Twitter Tips Blog: TwiTip?
about 1 year ago - 7 comments
Problogger Darren Rowse has launched a blog devoted to all things Twitter. He launched it just days ago and in it he sets out to cover tips on how to best use the micro blogging platform. TwiTip is a blog about Twitter. It’s a place that I’ll be posting tips, news and reviews of tools
Hot NEW! Contest For Bloggers- Win Big! (and help me in the process)
about 1 year ago - 14 comments
A friendly blogger by the name of Max, who runs the Villiat Business Blog has created an awesomerific contest to help yours truly. Read on below for the juicy details. The contest is basically a (feedback initiative) to get YOU to help ME and WIN some cool prizes in the process. As some of you
about 1 year ago
Hello Missy,
Thanks for the mention, I appreciate it. RE: The URL’s blocked on your robots file just look at your robots file and see what pages are blocked.
You can see what your robots file says in GWT, but you can only edit it from your hosting files. Usually a robots.txt file is created by the webmaster, so if you did not create it, I would certainly look into what is in yours and edit it correctly.
There are pages that you do not want indexed and that you do not want Google to crawl to preserve link juice and crawl juice for the important pages. You do not want for instance, your css files, wp-admin, feeds, about, contact pages etc…to be crawled and indexed. You need to block categories and tags on blogs because these would be considered duplicate content, all this is accomplished with a robots file.
Here is my robots file, you can copy it and use that for your blog, edit out pages you may not have as needed, such as terms-of-service, that you may not have on your blog.
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /*.css$
Disallow: /cont/
Disallow: /scripts/
Disallow: /wp-content/themes/
Disallow: /?s=
Disallow: /rss/
Disallow: /feed/
Disallow: /feed/
Disallow: /terms of use/
Disallow: /disclaimer/
Disallow: /privacy policy/
Disallow: /search/
Disallow: /about/
Disallow: /contact me/
Disallow: /tags/
Disallow: /archives/
Disallow: /category/
Disallow: /trackback/
Disallow: /contact-form/
Disallow: /comments/feed/
Disallow: /*.avi$
Disallow: /*.cgi$
Disallow: /index.php
Disallow: /*.js$
Disallow: /*.inc$
User-agent: Mediapartners-Google*
The 19 not found URL’s may be redirects, what are they? and What is the reason that GWT is showing them not found is a good place to start.
about 1 year ago
Hey, thanxs JR. I think i better understand now what is the robots.txt file. (or atleast its purpose)
You say it needs to be changed via my host, which file is it in?
Thanxs for the copy of your robots file, will look it over. That is awesomesauce!
As for the 19 URL’s not found, i will have to look into it. I honestly don’t know.
Thanxs for your groovy help. Appreciate it loads.
about 1 year ago
Hey Missy
I agree the article by JR on Google’WMT is awesome. On last check, you don’t have a robots.txt file yet which is not an ideal situation for you. I guess JR has explained to you what it’s for….
The other 19 urls could be some of the broken links within your site that you need to fix. Use a broken link checker plugin to see where the link is and rectify it immediately.
Hope it helps…
Yan
Blog for Beginners’s last blog post..Me, Carl Ocab and Direct Response Templates