[Jason]

Many bloggers do not use robots.txt file at all. Even if they use robots.txt, there are only few lines of code to restrict search engines bot. As many of you already know robots.txt file analysis is one of the essential factor for search engine optimization. Google also included this tool inside Google Webmaster tools to [...]

To view links or images in this forum your post count must be 1 or greater. You currently have 0 posts.




To view links or images in this forum your post count must be 1 or greater. You currently have 0 posts.

[gingercaily]

robot.txt is use to tell the search engine robots not to crawl certain
folders and pages of your web site.

[zenneth]

People as I know use the robot.txt file to stop the search engines from crawling certain web pages. It's like an instruction file to let the bots and spiders know what they can index and what they can't.