robots.txt

[h1sks]

Hello Everyone

What will the following robots.txt file do?

[Hosting]

Hello Everyone

What will the following robots.txt file do?

And where's the following robots.txt file ? :rolleyes:

[Raju]

This file is used to hide the irrelevant web pages of our website from search engine's crawler.

[RH-Calvin]

robots.txt file is basically used to hide some pages of your website from search engine crawling. You can feed the web pages of your website which you do not want search engines to find. But these pages will be visible to your visitors.

[thomosmax]

The robots txt file is used to hide the irrelevant content or web page from crawler.

[anirban09P]

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

[jainhost]

When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html or index.php page. It is your robots.txt file.

[milos87popovic]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

[john00]

If you want to crawl everything on your website. You do not need any robots.txt file.

[gamsoftware]

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. It is by no means compulsory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

[BenWood]

Thanks for sharing..............

[Macwong]

Robot.txt file mainly used to protect the web directories, images and all the files that you do not want to crawl by the search engine.
Format of Robot.txt is

user-agent: *
disallow: /images/

Where user-agent means for witch search engine you want to hide the files or pages where " * " used for all search engine..
If you want to disallow only from the Google-bot you robot.txt file will be

user-agent: gogolebot
disallow: /images/

Disallow means which directory or file you are going to hide from the search engine.
in the above text we disallow the image directory from Google search engine.

For more practice by yourself. Thanks.

[Neo.Reeves]

This file is used to hide the irrelevant page of your website from crawler...

[jayanta1]

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file.

[JessicaJohn]

In this file to tell the Google spider that which links you don't want to allow it, it is very important and helpful file to disallow the particular pages. If you don't put anything in this file its mean you don't want to disallow any page.

[jackthomas087]

Reboot Txt File .

The /robots.txt is a de-facto standard, and is not owned by any standards body. There are two historical descriptions:

the original 1994 A Standard for Robot Exclusion document.
a 1997 Internet Draft specification A Method for Web Robots Control..

[arianagrand]

Thank for the sharing useful information ..

[Teamwork]

robots.txt file is a part of seo.

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

[Raju]

Robots.txt is used to gives search engines information about the pages a company wants indexed or crawled. You can find this page by doing to YOURDOMAIN/robots.txt.

[amit45]

Hello Friends,
RObot.txt file is a file which save in root directory. Robot.txt tells search engine engine crawler that which pages has to be index or not?

[ashiselma]

Robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

[StuartSpindlow1]

You can block a particular page of your website with robot.txt which you don't want to crawl by Google.