Blogger Custom Robots.txt for Better SEO

Robots.txt is a text file that controls web crawling robots (also known as search engine robots) and tell them which page they should crawl and which they should not, which web directory they should crawl and which they should not, & also which links they should crawl and which they should not.

Blogger Custom Robots.txt for Better SEO

Why Robots.txt is so Important?

The simple answer to it is when it comes to website, SEO is the must do thing. To do SEO, adding right robots.txt is the necessary thing. So let's get better understanding to it.

A sample robots.txt file


User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search
Disallow: ?updated-max
Allow: / 

Sitemap: 
https://www.example.com/sitemap.xml

  • Here the file line (User-agent) is declaring the robot type which is Mediapartners-Google and it is set to none. It means that AdSense ads can appear
  • The next User-agent is set to * and Disallow are /search & ?updated-max and Allow is /. Here User-agent:* means all search engine bots are disallowed to /search & ?updated-max pages and allow are all pages except the disallowed pages.
  • The next thing is Sitemap which contains the sitemap of blog. Here comes the more tricky part. As you can see the default is set to https://www.example.com/sitemap.xml. In Blogger, it will only allow first 25 posts to be crawled. If you want more posts to be crawl then use different Sample codes given below.

Sample 1 - to submit only first 25 posts


User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search
Disallow: ?updated-max
Allow: / 

Sitemap: 
https://www.example.com/sitemap.xml

Sample 2 - to submit first 25 posts & also pages


User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search
Disallow: ?updated-max
Allow: / 

Sitemap: 
https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-pages.xml

If you need more pages to be crawled then use below sample.

Sample 3 - to submit first 500 posts & also pages


User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search
Disallow: ?updated-max
Allow: / 

Sitemap: 
https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.example.com/sitemap-pages.xml

Note: Don't forget to change example.com with your own URL.

Just use any sample which suits you most. We recommend you to use Sample 3 as it is best for big sites having work on both posts and pages.

That's it for this time. If you have any questions related to this then comment below. We'll gladly help you.