Robots.txt Features
- No follow, No Index Tag Option for each page, by default it is yes and can be overridden manually
-
Robots.txt, just ensure your CMS doesn't stop robots.txt from rendering because of some settings, make a basic robots.txt in the root folder and ensure it is rendered properly. Download the basic robots.txt folder.
Manage page views to search engine via robot.txt
The CMS has an in-built module which makes effortless creation of important files like robot.txt possible with just a check. The image below is the section and the user just has to check the box beside "allow search engine to index this page" to generate an automatic robots file. It also allows page specific customization - it gives you the option to disallow search engine to crawl, index and prevent caching the particular page. Advance protocols like preventing description from appearing on search result can also be done.
