Blogger Custom Robots.txt for Better SEO

  • Join us to download any resource for free. You will be able to download or guest post.
    Sign Up Now

kashif

Administrator
Staff member
VIP Member
Apr 12, 2022
5,398
207
63
Pakistan
www.vuinsider.com
Robots.txt is a text file that controls web crawling robots (also known as search engine robots) and tells them which page they should crawl and which they should not, which web directory they should crawl and which they should not, & also which links they should crawl and which they should not.

Blogger Custom Robots.txt for Better SEO

Why Robots.txt is so Important?​

The simple answer to it is when it comes to website, SEO is the must do thing. To do SEO, adding right robots.txt is the necessary thing. So let's get better understanding to it.

A sample robots.txt file


Code:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml



  • Here the file line (User-agent) is declaring the robot type which is Mediapartners-Google and it is set to none. It means that AdSense ads can appear
  • The next User-agent is set to * and Disallow are /search & ?updated-max and Allow is /. Here User-agent:* means all search engine bots are disallowed to /search & ?updated-max pages and allow are all pages except the disallowed pages.
  • The next thing is Sitemap which contains the sitemap of the blog. Here comes the more tricky part. As you can see the default is set to . In Blogger, it will only allow the first 25 posts to be crawled. If you want more posts to be crawled then use different Sample codes given below.


Sample 1 - to submit only first 25 posts​


Code:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml

Sample 2 - to submit first 25 posts & also pages​


Code:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-pages.xml

If you need more pages to be crawled then use below sample.

Sample 3 - to submit first 500 posts & also pages​


Code:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: ?updated-max
Allow: /

Sitemap:
https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.example.com/sitemap-pages.xml

Note: Don't forget to change example.com with your own URL.

Just use any sample which suits you most. We recommend you to use Sample 3 as it is best for big sites having work on both posts and pages.

That's it for this time. If you have any questions related to this then comment below. We'll gladly help you.
 

Similar threads

Staff online

Members online

Forum statistics

Threads
3,738
Messages
6,215
Members
5,615
Latest member
fffffer

About us

  • Our community has been around from 2017 and pride ourselves on offering unbiased, critical discussion among people of all different backgrounds. Providing Free Resources to help newcomers/startups. We are working every day to make sure our community is one of the best.

Quick Navigation

User Menu

AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features of our website. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock