Robots.txt
What is a robots.txt file?

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”).

In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents.

INSTALLATION ON 2.3.x & 3.x:
Upload dynamic_robots.ocmod.zip via Extension Installer
Override the files if exists
Refresh Modifications
Go to User Permissions and give the permissions for robots.php and related files
Go to Extensions -> Sitemap, Looks robots.txt and click install for every each store which you want to install dynamic robots.txt
Select Activated
Give time for expiring the caching (in seconds)
Add robots.txt rules in the textarea below
Click Save

INSTALLATION ON 2.1.x:
Upload robots_txt.ocmod.zip via Extension Installer
Refresh Modifications
Go to User Permissions and give the permissions for analytics/robots
Go to Extensions -> Analytics, Looks Dynamic Robots.txt Rules
Select Store
Select Activated
Give time for expiring the caching (in seconds)
Add robots.txt rules in the textarea below
Click Save

NOTES:
YES, has multistore support!! You can add or edit new and different rules for every each stores in your Opencart installation.

IMPORTANT:
BACKUP YOUR PREVIOUSLY robots.txt

VERY IMPORTANT:
Don't add sitemap rule in robots.txt !

SUPPORTED LANGUAGES:
BULGARIAN and English

FOR TESTING AFTER INSTALLATION AND CONFIGURATING, JUST OPEN YOUR STORE -> https://yourstore-name.tld/robots.txt

NOTES:
There isn't cache settings for 3th Opencart version!