Comments on: What went Wrong with My Robots.txt? http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/ Get the latest TechBeat! Sat, 22 Nov 2008 15:08:55 +0000 http://wordpress.org/?v=2.6.3 By: Narayanan Hariharan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-99 Narayanan Hariharan Mon, 04 Jun 2007 00:03:15 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-99 @ Nathan and Vinod, Thanks for the help guys! @ Nathan and Vinod,

Thanks for the help guys!

]]> By: Vinod http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-98 Vinod Sat, 02 Jun 2007 10:12:02 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-98 Hari.. you can try copying my robots.txt. its kinda optimized. Make sure that you've got the Sitemap generator also installed. Currently I have the sitemap generated only for the posts, if i include a lot more like archives, categories etc.. there have been an increase in supplemental results. Hari.. you can try copying my robots.txt. its kinda optimized.
Make sure that you’ve got the Sitemap generator also installed.
Currently I have the sitemap generated only for the posts, if i include a lot more like archives, categories etc.. there have been an increase in supplemental results.

]]> By: Nathan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-96 Nathan Fri, 01 Jun 2007 23:29:37 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-96 It's really hard to say why he added a couple hundred pages to the index and there is no many pages to really track down where the issue is. No matter what john does he's going to have at average of 2-3 time more pages in the index than he actually has posts because he uses multiple category tags. It’s really hard to say why he added a couple hundred pages to the index and there is no many pages to really track down where the issue is. No matter what john does he’s going to have at average of 2-3 time more pages in the index than he actually has posts because he uses multiple category tags.

]]> By: Narayanan Hariharan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-95 Narayanan Hariharan Fri, 01 Jun 2007 20:21:57 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-95 Oh! I wasn't aware of that. BTW any clue how/why John Chow's Supplemental Index had such a steep increase in the 7 to 8 days it has been up? Oh! I wasn’t aware of that.
BTW any clue how/why John Chow’s Supplemental Index had such a steep increase in the 7 to 8 days it has been up?

]]> By: Nathan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-94 Nathan Fri, 01 Jun 2007 17:38:15 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-94 You've got to wait for the major index update every 36 days or so. That's when you'll see pages come off the index. http://www.metamend.com/google-dance.html You’ve got to wait for the major index update every 36 days or so. That’s when you’ll see pages come off the index.

http://www.metamend.com/google-dance.html

]]> By: Narayanan Hariharan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-93 Narayanan Hariharan Fri, 01 Jun 2007 16:45:22 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-93 Hi Nathan, The Google Bot cached my website on the May 27th. But there was no change. If you see the screenshot above, there has been a caching on May 30th also. Again no change. Will make the categories display an excerpt. Thanks for the suggestion. :) Hi Nathan,

The Google Bot cached my website on the May 27th. But there was no change. If you see the screenshot above, there has been a caching on May 30th also. Again no change.
Will make the categories display an excerpt. Thanks for the suggestion. :)

]]> By: Nathan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-92 Nathan Fri, 01 Jun 2007 16:00:37 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-92 You need to make you categories disply an excerpt as opposed to the entire post like you currently have it setup. That'll kill tons of your supplemental index problems. You need to make you categories disply an excerpt as opposed to the entire post like you currently have it setup. That’ll kill tons of your supplemental index problems.

]]> By: Nathan http://www.techspikes.com/2007/06/what-went-wrong-with-my-robotstxt/#comment-91 Nathan Fri, 01 Jun 2007 15:57:15 +0000 http://www.techspikes.com/2007/06/01/what-went-wrong-with-my-robotstxt/#comment-91 Not sure why you are seeing a drop in traffic. If your robots.txt file is messed up you'd have seen a ton of your pages become uncached. Changing your robots.txt file won't effect your site overnight. You have to wait for the googlebot to recache your entire site which could take 2-3 weeks. Not sure why you are seeing a drop in traffic. If your robots.txt file is messed up you’d have seen a ton of your pages become uncached.

Changing your robots.txt file won’t effect your site overnight. You have to wait for the googlebot to recache your entire site which could take 2-3 weeks.

]]>