Robots.txt Configuration in Magento 2. Nofollow and Noindex

Magento 2 Robots.txt

Robots exclusion standard, also known as "robots.txt file", is important for your website or store when communicating with search engines crawlers. This standard defines how to inform the bots about the pages of your site that should be excluded from scanning or, visa versa, opened for crawling. That’s why robots.txt file is significant for the correct website indexation and its overall search visibility.

Magento 2 store owners utilize the **robots.txt** file to

• Exclude potential duplicate content: Magento 2 sites can prevent search engine penalties caused by redundant content using the `Disallow:` directive in robots.txt.
• Ensure privacy for admin or confidential directories: safeguard confidential Magento 2 directories from indexing.
• Prevent indexing of sections under development or testing.
• Server Strain: Restrict bots from heavy pages to minimize server loads.

Tips for Magento 2 Robots.txt file

• Always position the robots.txt file in the root directory.
• For Magento 2 stores utilize the platform's in-built settings to manage the robots.txt file with ease.
• Consistently review and modify the robots.txt file according to the changes in your Magento 2 store.

How to configure Robots.txt file in Magento 2

By default, Magento 2 allows you to generate and configure robots.txt files. You can prefer to use the default indexation settings or specify custom instructions for different search engines.

To configure the robots.txt file follow these steps:

1. Open Content tab, select Design option and select the Configuration.

2. Open the Search Engine Robots section and switch the Default Robots option to one from the drop-down.

3. Enter custom instructions to the robots.txt file.

4. Hit Save Config to complete the operation.

Examples of robots.txt file

We recommend using the following custom robots.txt for your Magento 2 store:

User-agent: *



# Directories

Disallow: /404/

Disallow: /app/

Disallow: /cgi-bin/

Disallow: /downloader/

Disallow: /errors/

Disallow: /includes/

Disallow: /lib/

Disallow: /magento/

Disallow: /media/captcha/



# Paths (clean URLs)

Disallow: /index.php/

Disallow: /catalog/product_compare/

Disallow: /catalog/category/view/

Disallow: /catalog/product/view/

Disallow: /catalog/product/gallery/

Disallow: /catalogsearch/

Disallow: /checkout/

Disallow: /control/

Disallow: /customer/

Disallow: /customize/

Disallow: /sendfriend/

Disallow: /ajaxcart/

Disallow: /ajax/

Disallow: /quickview/

Disallow: /productalert/

Disallow: /sales/guest/form/



# Files

Disallow: /cron.php

Disallow: /cron.sh

Disallow: /error_log

Disallow: /install.php

Disallow: /LICENSE.html

Disallow: /LICENSE.txt

Disallow: /LICENSE_AFL.txt

Disallow: /STATUS.txt

Disallow: /get.php



# Paths (no clean URLs)

Disallow: /*.php$

Disallow: /*?SID=

Disallow: /*PHPSESSID



Sitemap: https://www.example.com/sitemap.xml

Lets consider each groups of commands separately.

1.Stop crawling user account and checkout pages by search engine robot:

Disallow: /checkout/

Disallow: /onestepcheckout/

Disallow: /customer/

Disallow: /customer/account/

Disallow: /customer/account/login/

2.Blocking native catalog and search pages:

Disallow: /catalogsearch/

Disallow: /catalog/product_compare/

Disallow: /catalog/category/view/

Disallow: /catalog/product/view/

3.Sometimes Webmasters block pages with filters.

Disallow: /*?dir*

Disallow: /*?dir=desc

Disallow: /*?dir=asc

Disallow: /*?limit=all

Disallow: /*?mode*

4.Blocking CMS directories.

Disallow: /app/

Disallow: /bin/

Disallow: /dev/

Disallow: /lib/

Disallow: /phpserver/

Disallow: /pub/

5.Blocking duplicate content:

Disallow: /tag/

Disallow: /review/

6.Add the following code to the robots.txt file in order to hide specific pages:

User-agent: *

Disallow: /myfile.html

6.Don’t forget about domain and sitemap pointing:

Host: (www.)domain.com

Sitemap: http://www.domain.com/sitemap_en.xml

Meta robots tags: NOINDEX, NOFOLLOW

After configuring the robots.txt file, you can switch your attention to Nofollow and Noindex meta robots tags.

Meta robots tags are important for Magento 2 stores aiming for advanced control over page indexing. Positioned within a page's `` section, these tags tell search engines precisely how to treat individual pages, distinguishing them from the broad directives of the robots.txt file.

Types of Magento 2 Meta Robots tags

• noindex: perfect for Magento 2 pages you don't wish to index.
• nofollow: directs search engines to ignore the links on specific Magento 2 pages.
• noarchive: prevents caching of your Magento 2 page by search engines.
• nosnippet: ensures no part of the Magento 2 page appears as a snippet in search results.

How to configure the meta robots tags in Magento 2

To apply Nofollow or Noindex to your current configuration you can either update the robots.txt file or use meta name=“robots” tag.

Some possible combinations:

meta name="robots" content="index, follow" 

meta name="robots" content="noindex, follow" 

meta name="robots" content="index, nofollow" 

meta name="robots" content="noindex, nofollow"

Important notice:

Tags Noindex and Nofollow has many advantages over blocking a page through robots.txt:

1. Robots.txt prevents a page crawling during scheduled website crawling. However, this page could be found and crawled from other websites links.
2. If a page has inbound links all juice will be transmitted to other website pages through this page internal links.

Mageworx SEO Suite extension

Using the instruction above, you will be able to manually configure the robots.txt file of your Magento 2 store and hide the unnecessary parts of code or spread the weight of pages. However, it's very important to review the changes regularly and adjust the meta robots configuration. Mageworx Magento 2 seo plugin provides advanced tools to do this for you:

• Set up meta robots for any product page or for products in bulk
• Set up meta robots for any category page
• Configure the meta robots for any CMS page
• Configure meta robots tag for any custom page in your Magento 2 store
• Advanced meta robots tag setup for layered navigation pages (for any attribute combinations, for category filter pages with multiple selections)
• and many other