Robots.txt

Category: On-site Value

Importance Level: 2/5

 

• Do’s: Have the XML sitemap link and to block the unwanted pages from it. Do give a validation checks for different type of URLs.
• Don’ts: Don’t go too crazy about it :)