Which is better Robots.txt or Meta Robots Tag?

[ninadordev]

Hello friends,

Which is better Robots.txt or Meta Robots Tag?

[Neo_5678]

Which one have you employed currently?

[mtgfanboy]

It depends on your choice which one suits you better like if you want to deindex a page or directory from Google’s Search Results then I suggest that you use a “Noindex” meta tag rather than a robots.txt whereas Robots.txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages.

[MVMInfotech]

Personally I think the meta tag is even more useful than the robots.txt file.

The robots.txt file is intended to PREVENT bots from spidering files in the first place.

The robots meta tag can in some ways be used to get indexed files deindexed. And it
is more versatile as a tool to 'guide' bots through your website without having them
index all the files.

[dennis123]

On the most basic level, neither the meta robots tag or the Robots.txt has authority over the other – but rather the “noindex” request has authority over the “index” request.

What I’ve suggested to development teams I’ve worked with in the past is to set the Robots.txt file to allow all crawlers, then use the meta robots tag to request that a page not be indexed on a page by page basis. This is easy to remember, easy to update, and makes it easy to avoid conflicts and confusion.

[raheel]

This is Robots file

User-agents:*
Disallow :

This is Meta Robots tag

<meta name="robots" content="noindex,nofollow"/>

[RH-Calvin]

Robots.txt is better compared to meta robots tag. It is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[pharmasecure]

Meta Robots tag is much better as it helps in forcing the search engine crawlers not to index and display the hidden pages in your server.

‎

[Jessicad0505]

Robots.txt is better in my opinion.

[Prateektechnoso]

Robots.txt and robots tag both are equal level to contro your Website URL's to crawl by Search Engine Spider or Crawler.


For your information:

Prateektechnosoft is a Netsuite Partner and expertise in NetSuite ERP, CRM, Cloud CRM, PSA and other Netsuite Solutions. And also providing Netsuite services of implementation, integration, support & development services.

[hiren_popat]

Robots.txt:
it gives instruction to search engine spider that which pages of your sites you want to index and which pages want to ignore, there are many pages in website but not all useful for users so it’s not important to index those pages in search engine, A robots.txt file tell to the robots which pages or parts of the website that should be index and which parts should not.
an example of Robots.txt file is below:
“(!) User-agent: Googlebot/MSNbot/yahoobot.
Disallow: add your link here which do not want to index”.

“(!!) User-agent:*
Disallow: /”

“(*) indicates all the search engines.
(/) indicates all pages of a site.”

you can inform all types of search engine robots for indexing status in a time.

Meta Robots:
it is an HTML tag it gives a command to the search engine that what to do with your pages, whether pages consider for crawl and index or not, there are two types of activities in meta robots.
<meta name=”robots” content=” index,noindex“/>
<meta name=”robots” content=” follow,nofollow“/>

* follow/no follow: it gives a command to crawler/spider about follow or no follow links of a web page.

both are good for preventing page to index or ignore.
another use of meta robots is that no-follow link means it prevents pass link juice to another site.

[Preeju M S]

Both Robots.txt and Meta Robots Tag are important.