Yes we needed robot.text for crawling of my website page.
Yes we needed robot.text for crawling of my website page.
Robots.txt file on a website will function as a request that specified robots, ignore specified files or directories when crawling a site. So it is very important.
|
Bookmarks