Blogger Custom Robots.txt for Better SEO

[kashif]

Robots.txt is a text file that controls web crawling robots (also known as search engine robots) and tells them which page they should crawl and which they should not, which web directory they should crawl and which they should not, & also which links they should crawl and which they should not.

Why Robots.txt is so Important?​

The simple answer to it is when it comes to website, SEO is the must do thing. To do SEO, adding right robots.txt is the necessary thing. So let's get better understanding to it.

A sample robots.txt file

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml

• Here the file line (User-agent) is declaring the robot type which is Mediapartners-Google and it is set to none. It means that AdSense ads can appear
• The next User-agent is set to * and Disallow are /search & ?updated-max and Allow is /. Here User-agent:* means all search engine bots are disallowed to /search & ?updated-max pages and allow are all pages except the disallowed pages.
• The next thing is Sitemap which contains the sitemap of the blog. Here comes the more tricky part. As you can see the default is set to
  https://www.example.com/sitemap.xml
  . In Blogger, it will only allow the first 25 posts to be crawled. If you want more posts to be crawled then use different Sample codes given below.

Sample 1 - to submit only first 25 posts​

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml

Sample 2 - to submit first 25 posts & also pages​

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-pages.xml

If you need more pages to be crawled then use below sample.

Sample 3 - to submit first 500 posts & also pages​

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.example.com/sitemap-pages.xml

Note: Don't forget to change example.com with your own URL.

Just use any sample which suits you most. We recommend you to use Sample 3 as it is best for big sites having work on both posts and pages.

That's it for this time. If you have any questions related to this then comment below. We'll gladly help you.

 

[startoblog]

Hi.. Tyvm for u tips i can try this method...

 

[irfanshah]

YOU CAN GENERATE FOR BLOGGER.COM POSTS AND PAGAES https://www.marwat-tech.com/p/best-blogger-sitemap-generator.html

 

[rramdandi]

if your article has thousands use this

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow : /

Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=501&max-results=500
Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=1001&max-results=500
Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=1501&max-results=500
Sitemap: https://www.example.com/sitemap-pages.xml

 

[matzy]

Useful resource