What is a robots.txt file?

[ryansmith]

What is a robots.txt file?

I don't know therefore let me know.

[RH-Calvin]

Robots.txt is a text file that is used to define crawler activity on your website. It list the webpages that need to be disallowed from crawling.

[Joseph f]

Robots.txt to tell google which file should index and which file should not index.
Below tha code of robotx.txt

User-agent: [the name of the robot the following rule applies to]

Disallow: [the URL path you want to block]

Allow: [the URL path in of a subdirectory, within a blocked parent directory, that you want to unblock]

[edwina]

Robot.txt:
It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

[smmexpertss]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html.

[jane1]

This is search engine directive file that is used for preventing search engines not to crawl a website.

[ryansmith]

Thanks all for providing such valuable information about robots.txt file. Indeed, I was very curious about this and now so much interested in it.

[seoservice7]

Robot.txt is a text file that permits the google bot what pages should he crawled and what pages he shouldn't.
User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

[kevinbelgiean]

Robots.txt is a text document file which is mainly used to give instructions for search engine robots to crawl/not crawl the particular page/website.

[ramskl]

The robots.txt file is a very powerful file if you're working on a site's SEO, but one that also has to be used with care.

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

[CarolineMurphy]

Robots.txt is a text file which you place on your site root folder to tell Google bot which pages you would like them not to crawl. It shows like below:

User agent: *
Allow: /
Disallow: /admin

Note: Put Allow which you want google bot crawl. In the other hand, You have to place disallow if you don't want them to crawl.

[KateSteele]

Robots.txt file inform the search engine crawler which page or folder has to crawl and which folder has not to crawl. If you specify data for all bots then, you need to put (*) and the data for specific bot for instance (google bot) then the specific bot commands will be followed, Below I mentioned it, just go through it.

User-agent: *
Disallow: /admin

and the second one for google bot

User-agent: googlebot
Disallow: /se/

[dzeetal]

robots.txt is file which is allow to web page index or not index, follow or nofollow.

[michaelclarke]

Robots.txt is use instructions about their site to web page index or not index this is called The Robots.