How to create robots.txt in wordpress

Robots.txt file is extremely important for WordPress Website, As Robots.txt file decides whether your website is going to be crawled by Search Engine crawler or not. If yes then how & If no then how?

Robots.txt file will affect your crawling & ranking status, so it becomes very important that if you have just updated anything on your website & that must be crawled instantly by the Search Engine Crawler. so that your content can be indexed. It’s one of the major things to be done in SEO.

Note:- Crawling Speed = Indexing Speed

The sole thing is, what ever you update, post and write on your web page that must be crawled by Search Engine crawler to get indexed.

Just keep in mind each website has a default Robots.txt & you know we always need to customize a default thing as our own. So, here I’ll provide a customized Robots.txt and you just need to copy & paste that.

robots.txt file

What is Robots.txt

Firstly have a look at Search Engine Crawler/bot/spider, A search engine bot has a job assigned of crawl & index. As a Search Engine crawler visits your website, crawler follows all the links on your website page & then make it indexed on Search Engine. Finally, you see your website is indexing on Search Engines(Google, Bing, etc.)

By name it’s clear, this file is for robots. Robots.txt file always locates in root directory of a website & instructs Crawler to crawl a website, as the rules mentioned in the robots.txt file. Means crawler/bot/spider blindly follows the instruction of robots.txt file.

There are lots of things on a website like, tags, comments, category, wp-admin like unnecessary pages which don’t add any value to the users so they must not be indexed on SERP directly & this is what a robots.txt file set a rule.

Attention:- A single change or mistake in robots.txt will remove you from being indexed on a Search Engine so, be careful. Just follow as mentioned.

Don’t forget to check these articles:-

How to Update robots.txt file in WordPress

There are many ways to update your robots.txt file. You can update it from your hosting or your favorite one from the WordPress Dashboard.

Here we go with WordPress Dashboard as it’s the simplest & safe method to update it.

Let’s see how you can create a customized fast crawl able & index able robots.txt file.

Step 1:- For this you should have a Yoast SEO Plugin user. As shown in image go to yoast seo plugin icon & hit on tools.

Step 2:- Now you will see a page something like the image below. Just hit on edit & you’ll be redirected to edit page.

Step 3:- Now hit on create robots.txt file. You will see this page. Here just select all & delete everything but don’t hit on save changes.

Step 4:- Copy & Paste the text given below in the file box & hit on save changes now.

Robots.txt file download link

Note:- Make sure to remove your site url text with the url of your site.

After save changes you will have a fully optimized robots.txt file, which will instruct the crawler what to crawl & what not to crawl. This will help your site to be crawl fast & index fast.

I hope this will help you to index fast on Google & this is all about create robots.txt file in WordPress. For I’ll try to create another one post or you can visit this link.

Don’t forgot to leave a comment, if this article has helped you a bit.

Thank you for being here.

Leave a comment