Printable View
The robots.Txt file is a part of the REP, a group of web necessities that control how robots crawl the web, access and index content material, and serve that content material up to users. These crawlers advised to disallow or allow the user agents.
It is not necessary in your website.
Yes it is essential, as it is a component of on page optimization, so it is a must do thing to add robots file in your website, with the help of robots file you can control deindexing of any pages and can tell google and other search engine bots that which directories are private and should not be indexed.
Robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.