WordPress Robots.txt optimization – Website traffic, ranking & SEO Booster + Woocommerce + Growth hacking tools

Description

Better Robots.txt creates a virtual robots.txt, helps you boost your website SEO (indexing capacities, Google ranking,etc.) and your loading performance –100% compatible with YOAST SEO & WooCommerce

With Better Robots.txt, you can identify which search engines are allowed to crawl your website (or not), specify clear instructions aboutwhat they are allowed to do (or not) and define a crawl-delay (to protect your hosting server against aggressive scrapers). Better Robots.txt also gives you full control over the robots.txt file content via the custom setting box.

A quick overview:

SUPPORTED IN 6 LANGUAGES

Better Robots.txt plugins are translated and available in: Chinese –汉语/漢語, English, French – Français, Russian –Руссɤɢɣ, Portuguese – Português, Spanish – Español.

Did you know that…

  • The robots.txt file is a simple text file placed on your web server which tells web crawlers (like Googlebot) whether they should access a file.
  • The robots.txt file controls how search engine spiders see and interact with your web pages;
  • This file and the bots they interact with are fundamental parts of how search engines work;
  • The first thing a search engine crawler looks at when it is visiting a page is the robots.txt file;

The robots.txt is a source of SEO juice just waiting to be unlocked. Try Better Robots.txt !

About the Pro version (additional features):

1. Boost your content on search engines with your sitemap !

Make sure your pages, articles, and products, even the latest, are taken into consideration by search engines !

The Better Robots.txt plugin was made to work with the Yoast SEO plugin (probably the best SEO Plugin for WordPress websites). It will detect if you are currently using Yoast SEO and if the sitemap feature is activated. If it is, then it will add instructions automatically into the Robots.txt file asking bots/crawlers to read your sitemap and check if you have made recent changes in your website (so that search engines can crawl the new content that is available).

If you want to add your own sitemap (or if you are using another SEO plugin), then you just have to copy and paste your Sitemap URL, and Better Robots.txt will add it into your Robots.txt file.

2. Protect your data and content

Block bad bots from scraping your website and commercializing your data.

The Better Robots.txt plugin helps you block most popular bad bots from crawling and scraping your data.

When it comes to things crawling your site, there are good bots and bad bots. Good bots, like Google bot, crawl your site to index it for search engines. Others crawl your site for more nefarious reasons such as stripping out your content (text, price, etc.) for republishing, downloading whole archives of your site or extracting your images. Some bots were even reported to pull down entire websites as a result of heavy use of broadband.

The Better Robots.txt plugin protects your website against spiders/scrapers identified as bad bots by Distil Networks.

3. Hide & protect your backlinks

Stop competitors from identifying your profitable backlinks.

Backlinks, also called “inbound links” or “incoming links” are created when one website links to another. The link to an external website is called a backlink. Backlinks are especially valuable for SEO because they represent a “vote of confidence” from one site to another. In essence, backlinks to your website are a signal to search engines that others vouch for your content.

If many sites link to the same webpage or website, search engines can infer that the content is worth linking to, and therefore also worth showing on a SERP. So, earning these backlinks generates a positive effect on a site’s ranking position or search visibility. In the SEM industry, it is very common for specialists to identify where these backlinks come from (competitors) in order to sort out the best of them and generate high-quality backlinks for their own customers.

Considering that the creation of very profitable backlinks for a company takes a lot of time (time + energy + budget), allowing your competitors to identify and duplicate them so easily is a pure loss of efficiency.

Better Robots.txt helps you block all SEO crawlers (aHref, Majestic, Semrush) to keep your backlinks undetectable.

4. Avoid Spam Backlinks

Bots populating your website’s comment forms telling you ‘great article,’ ‘love the info,’ ‘hope you can elaborate more on the topic soon’ or even providing personalized comments, including author name are legion. Spambots get more and more intelligent with time, and unfortunately, comment spam links can really hurt your backlink profile. Better Robots.txt helps you avoid these comments from being indexed by search engines.

5. Seo tools

While improving our plugin, we added shortcut links to 2 very important tools (if you are concerned with your ranking on search engines): Google Search Console & Bing Webmaster Tool. In case you are not already using them, you may now manage your website indexing while optimizing your robots.txt ! Direct access to a Mass ping tool was also added, allowing you to ping your links on more than 70 search engines.

We also created 4 shortcut links related to the best Online SEO Tools, directly available on Better Robots.txt SEO PRO. So that, whenever you want, you are now able to check out your site’s loading performance, analyze your SEO score, identify your current ranking on SERPs with keywords & traffic, and even scan your entire website for dead links (404, 503 errors, …), directly from the plugin.

6. Be unique

We thought that we could add a touch of originality on Better Robots.txt by adding a feature allowing you to “customize” your robots.txt with your own unique “signature.” Most major companies in the world have personalized their robots.txt by adding proverbs (https://www.yelp.com/robots.txt), slogans (https://www.youtube.com/robots.txt) or even drawings (https://store.nike.com/robots.txt – at the bottom). And why not you too? That’s why we have dedicated a specific area on the settings page where you can write or draw whatever you want (really) without affecting your robots.txt efficiency.

7. Prevent robots crawling useless WooCommerce links

We added a unique feature allowing to block specific links (« add-to-cart », « orderby », « fllter », cart, account, checkout, …) from being crawled by search engines. Most of these links require a lot of CPU, memory & bandwidth usage (on hosting server) because they are not cacheable and/or create « infinite » crawling loops (while they are useless). Optimizing your robots.txt for WooCommerce when having an online store, allows to provide more processing power for pages that really matter and boost your loading performance.

8. Avoid crawler traps:

« Crawler traps” are a structural issue within a website that causes crawlers to find a virtually infinite number of irrelevant URLs. In theory, crawlers could get stuck in one part of a website and never finish crawling these irrelevant URLs. Better Robots.txt helps prevent crawler traps which hurt crawl budget and cause duplicate content.

9. Growth hacking tools

Today’s fastest growing companies like Amazon, Airbnb and Facebook have all driven breakout growth by aligning their teams around a high velocity testing/learning process. We are talking about Growth Hacking. Growth hacking is a process of rapidly experimenting with and implementing marketing and promotional strategies that are solely focused on efficient and rapid business growth. Better Robots.txt provide a list of 150+ tools available online to skyrocket your growth.

More to come as always …

Captures d’écran

  • Better Robots.txt Settings Page
  • Better Robots.txt Settings Page
  • Better Robots.txt Settings Page
  • Better Robots.txt Settings Page
  • Robots.txt file output

Installation

Installing manually

  1. Unzip all files to the /wp-content/plugins/better-robots-txt directory
  2. Log into WordPress admin and activate the ‘Better Robots.txt’ plugin through the ‘Plugins’ menu
  3. Go to « Settings > Better Robots.txt » in the left-hand menu to start work on robots.txt file.

FAQ

Better Robots.txt plugin is enabled but why can’t i see any changes in robots.txt file?

Better Robots.txt creates a virtual robots.txt file. Please make sure that your permalinks are enabled from Settings > Permalinks. If permalinks are working then make sure that there is no physical robots.txt file on your server. Since it can’t write over physical file, so you must connect to FTP and rename or delete robots.txt from your domain root directory. It usually in /public_html/ folder on cPanel hostings. If you can’t find your domain root directory, please ask your hosting provider for help. If issue persists after taking these measures, please post it in support section or send a message to support@better-robots.com

Will there be any conflict with robots.txt which i’m already using?

If you have a pshysical robots.txt on your web hosting server, then this plugin will not work. As mentioned, it creates a virtual robots.txt file. Please follow steps in above answer if you want to use robots.txt file with this plugin.

How to add sitemap in robots.txt?

This feature is allowed in Better Robots.txt Pro version, which automatically add sitemap in robots.txt file. It detects sitemap from Yoast SEO plugin. In-case you’re using a different sitemap plugin or a manually generated sitemap then you can simply add sitemap URL in sitemap input field. If Yoast XML sitemaps are also enabled then you need to disable it first by simply going to Yoast General Settings > Features and disable XML Sitemaps feature.

Why should I optimise the robots.txt?

Why not? Considering that the robots.txt is the very first file read when your website is loaded by a browser, why not enable crawlers to continuously index your content? The simple fact of adding your Sitemap in the Robots.txt is simply common sense. Why? Did you list your website on Google Search Console, did your webmaster do it? How to tell the crawlers that you have new content available for indexation on your website? If you want this content to be found on search engines (Google, Bing, …), you have to have it indexed. That’s exacly what this instruction (adding the sitemap) aim to. One last point. The main reason why this plugin exists is due to the fact that 95% of the time (based of thousands of SEO analysis), the robots.txt is either missing, empty or misued, simply because it is either misunderstood or forgotten. Imagine now if it was activated and fully functional.

How can this plugin boost my website ranking?

Actually, this plugin will increase your website indexation capacity which leads to improve your ranking on Google. How ? Well, the idea of creating this plugin was taken after making hundreds of SEO optimization on professional and corporative websites. As mentioned before, 95% of analysed websites did not have what we could call an « optimized » robots.txt file and, while we were optimizing these websites, we realized that simply modifying the content of this file was actually « unlocking » these websites (based on daily SEMrush analysises). As we were used to working in 2 steps (periods of time), starting with this simple modification was already generating a significant impact on Google Ranking, and this, even before we started deeply modifying either the content, the site arborescence or META Data. The more you help search engines at understanding your website, the better you help your capacity of getting better results on SERPs.

Avis

Helpful, user-Friendly and great support !

It’s very rare that I take the time to review a plugin but in this case, I’ll make an exception because the team who made it really deserves a solid handshake.

It’s a great plugin, user-friendly (which is a BIG PLUS compared to a lot of other plugins – We are not all EXPERTS) and … Probably the most important to me, they provide a very effective support (and they really take the time to help my with my 3 websites).

Thank you so much for such a quality tool.

Joe

Just learn to copy and paste!

This plugin is somewhat nice looking and allow you to change your robots file with a little interface, but why would you add another plugin to cluster your WP Admin? Just upload a robots.txt file at the root of your website and copy/paste your robots info in it.

Saved my SEO!

Better Robots.txt saved me. For an unknown reason one of my sites had a « hook » at  » Discourage search engines from indexing this site » under Setting/Reading in admin panel. I had no idea about this and fought for hours to find why Google and the othersa did not index my site. I found no Robots.txt showing why.

Then I discovered Better Robots.txt. It found a file, even though it could not locate it. This robots.txt file stopped indexing. But in cooperation with their support and changing my SEO solution to Yoast it now seems that my site will get higher ranking than ever before! Thank you Gautier Dorval and the team at Pagup in Canada!

NB! I have no affiliation with Pagup. They just have a great product and support

Easy to Use and Time Saver

I have been modifying robots.txt files manually for years but this plugin is a real time saver. Better Robots.txt create robots.txt file with simple clicks even new user can easily use it.

Lire les 6 avis

Contributeurs & développeurs

« WordPress Robots.txt optimization – Website traffic, ranking & SEO Booster + Woocommerce + Growth hacking tools » est un logiciel libre. Les personnes suivantes ont contribué à cette extension.

Contributeurs

“WordPress Robots.txt optimization – Website traffic, ranking & SEO Booster + Woocommerce + Growth hacking tools” a été traduit dans 1 locale. Remerciez l’équipe de traduction pour ses contributions.

Traduisez « WordPress Robots.txt optimization – Website traffic, ranking & SEO Booster + Woocommerce + Growth hacking tools » dans votre langue.

Le développement vous intéresse ?

Parcourir le code, consulter le SVN dépôt, ou s’inscrire au journal de développement par RSS.

Journal

1.0.0

  • Initial release.

1.0.1

  • fixed plugin directory url issue
  • some text improvements

1.0.2

  • fixed some minor issues with styling
  • improved text and translation

1.1.0

  • added some major improvements
  • allow/off option changed with allow/disallow/off
  • improved overall text and french translation

1.1.1

  • fixed a bug and improved code

1.1.2

  • added new feature « Spam Backlink Blocker »

1.1.3

  • fixed a bug

1.1.4

  • added new « personalize your robots.txt » feature to add custom signature
  • added recommended seo tools to improve search engine optimization

1.1.5

  • added feature to detect physical robots.txt file and delete it if server permissions allows

1.1.6

  • added russian and chinese (simplified) languages
  • fixed bug causing redirection to better robots.txt settings page upon activating other plugins

1.1.7

  • added new feature: Top plugins for SEO performance
  • fixed plugin notices issue to dismiss for define period of time after being closed
  • fixed stylesheet issue to get proper updated file after plugin update (cache buster)
  • added spanish and portuguese languages

1.1.8

  • added new feature: xml sitemap detection
  • fixed translations

1.1.9

  • added new feature: loading performance for woocommerce

1.1.9.1

  • fixed a bug in disallow rules for woocommerce

1.1.9.2

  • boost your site with alt tags

1.1.9.3

  • fixed readability issues

1.1.9.4

  • fixed default robots.txt file issue upon plugin activation for first time
  • fixed php error upon saving settings and permalinks
  • refactored code

1.1.9.5

  • added clean-param for yandex bot
  • ask backlinks feature for pro users
  • avoid crawler traps feature for pro users
  • improved default robots.txt rules

1.1.9.6

  • added 150+ growth hacking tools
  • fixed layout bug
  • updated default rules