What Is Robot.Text File In Webmaster Tool?How To Use It?

                                                robots.txt file



1. What is the robots.txt file in webmaster tools?



a.Robots.text file is a file in your website by this file you can give a command to which part of your site you can't want to crawled by google bots or other search engines.There is specific kind of crawlers1.mobile crawler 2. desktop crawler.

2.non-image files



a.non-image files which are web pages robots.txt files also used by control the crawler traffic.You don't want to your server will down by over hauling of google crawler.

3.images



a.you can see when you search a query on google you cannot see any image with a blog post that reason robots.txt file prevents to showing images on the search result.

4.Blocking a URL



a.You can block a specific URL by robot.txt file

5.which part of your site you want to not crawled by bots?



a.The most important part of your site is you don't want to crawl is your website panel.If you crawl your admin panel then username will be displayed in search result.S o you need to paste this code in your robot.text file in your webmaster tools.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php



you can also block a url by this method just type



User-agent: *
Disallow:url

you can't stop to crawl no follow links to crawl bots will crawl but they not appear in search result.

No comments:

Powered by Blogger.