Robots.txt is a text file that instructs the web robots, i.e., search engines which pages to crawl and which not! Meta robots are a way for webmasters to offer search engines with details about their stores.
1. Importance of Robots.txt file in Magento 2 2. Steps to create Magento 2 Robots.txt
3. Meta robots tags
4. The recommended default settings for meta robots tags
5. Magento 2 robots.txt file examples
6. Common Web crawlersVisit
How to Configure Robots.txt and Meta Robots for Magento 2