Many bloggers do not understand the robots.txt option, and some blogger does not know the benefits of this. They left this option blank and this decrease the traffic of their site, and we all know the importance of visitors. To solve this problem today I am going to tell you, what is Robots.txt? How it works and how to attach this file to your website?
Robots.txt file plays a significant role in SEO of your blog. But if you do not know about this and you copy and paste this file from someone else’s website then it can harm your blog. So, before you activate it, you need to know the details of the robots.txt file.
Table of Contents
What is a Robot.txt file?
Each search engines have their Robots. You may be wondering; it is the robot which you have seen in the Hollywood movie. But that is not the robot we are talking about. It is a function of the search engines for crawling and indexing any website. You can allow or disallow to crawl and index any post to search engines using this file.
How robots.txt file works?
This file is similar to the traffic police. When you publish any new post and search engine bots come to your site, then the robots.txt tell the search bots to crawl and index that new post, so that new post comes quickly in the search result.
You will see this kind of code on the robots.txt file –
Most of the robots.txt files are like this. Now I am explaining these codes for better understanding.
User-agent:* This notify that you are ordering all kind of robots.
Disallow: /search This is telling to reject the keyword. In fact, this is telling not to index and crawl the search links of your blog.All labels of your blog include this search links. You do not need to crawl and index these tags.
Allow: / this code allows to index and crawls your homepage.
Sitemap: When you publish a new post then it will tell the search engine bots to crawl and index that new post.
How to insert a robots.txt file in Blogger?
- At first, Log in to blogger
- Then, go to your dashboard.
- Then Go to settings> search preferences.
- After that, Click on yes option of “Enable Custom Robots.txt” option
- Then copy-paste the above codes in the text box
- Save it.
Note: Replace “helperbus.com” by your website’s URL.
Thanks for reading this article. If you have any question, then comment us and if you like this then share it.