Months ago Blogger introduced custom robots.txt feature. This feature improve your blog's SEO but newbies really afraid to use this feature because it says :
Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.
But don't worry guys today we'll learn about this feature here on this article.
By default, every blog that uses the Blogger platform will have a robots.txt as follows:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.yourblog.com/feeds/posts/default?orderby=updated
As you can see in above robots.txt there are three parts of code and let me explain them to you.
Mediapartners-Google is a robot from Google Adsense, leave it because if you changed this then ads will not served will not fit with your content.
The next line is for all the robots and marked with an asterisk (*). On the default configuration, it is clear that the label of our blog is not indexed
Disallow: /search.
Keep in mind that a slash (/) is as your homepage, so for example if you want the label to get indexed, do not just fill up with a slash like this
Disallow: / because that would be you do not allow the robot tracing your blog.
For example if you want to block a robot for particular page ( for example contact page) you can simply write as follows:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /p/contact.html
Allow: /
Sitemap: http://www.yourblog.com/feeds/posts/default?orderby=updated
Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.
