Page 1 of 2 12 LastLast
Results 1 to 15 of 18
  1. #1
    Senior Member denniemark's Avatar
    Join Date
    Apr 2011
    Location
    Sky
    Posts
    195

    Post Is your website really needs robots.txt?

    We all know that before Google bot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. But Google webmasters suggest that if you want search engines to index everything in your site. You don't need a robots.txt file. But still lots of website webmaster use robots.txt, while they want search engines to index everything in your site. Google never suggest making a robots.txt file for every website, but webmasters creates.

  2. #2

  3. #3
    People use robot file to block certain page from crawling like those "under construction." If you want to make all the pages crawlable , then there is no need to make it.
    Last edited by Venus Brown; 09-04-2012 at 06:06 AM.

  4. #4
    Junior Member
    Join Date
    May 2012
    Posts
    5
    where did you read google do not suggest to create robots.txt file.Matt cutts suggest website should have robots.txt file.

  5. #5
    Registered User
    Join Date
    Aug 2012
    Posts
    82
    According to seo point of view robots.txt file should be there.

  6. #6
    Junior Member
    Join Date
    Aug 2012
    Posts
    18
    well as per latest updates no more use with them!!!!!!!!!!!!

  7. #7
    Registered User
    Join Date
    Jul 2012
    Posts
    186
    Hi,
    My website really needs robot.txt file that tell us which page you would like to spider or not.

  8. #8
    Registered User
    Join Date
    Aug 2012
    Posts
    16
    Robot.txt helps your website to suggest pages to crawl search engine. By SEO point of view Robot.txt needs.

  9. #9
    Registered User
    Join Date
    Aug 2012
    Posts
    31
    Hi,
    if you want to your website will or not each page then need robots.txt file

  10. #10
    Senior Member mikeslough5's Avatar
    Join Date
    May 2012
    Posts
    296
    Robot text file will use when you don't want that crawler don't see you specific page of your site.

  11. #11
    Senior Member
    Join Date
    Aug 2012
    Posts
    266
    If your site is very well structured and if you don't want search engines not to crawl any of your site pages, robot.txt will not be required for your site.

    I never use robot.txt in my site which i am optimizing.

  12. #12
    Senior Member denniemark's Avatar
    Join Date
    Apr 2011
    Location
    Sky
    Posts
    195
    Quote Originally Posted by Paul Jameson View Post
    Robot.txt helps your website to suggest pages to crawl search engine.
    It,s is wrong. Suppose your website is not have robots.txt file, then search engine will not crawl the website?

  13. #13
    Senior Member denniemark's Avatar
    Join Date
    Apr 2011
    Location
    Sky
    Posts
    195
    Quote Originally Posted by smitholivia View Post
    where did you read google do not suggest to create robots.txt file.Matt cutts suggest website should have robots.txt file.
    Read this Goolge Post (Read the 3rd paragraph from the Top): http://support.google.com/webmasters...&answer=156449

  14. #14
    Senior Member
    Join Date
    May 2012
    Posts
    200
    HI


    Yes every website need robot .txt for seo purpose.

  15. #15
    Senior Member
    Join Date
    May 2012
    Posts
    200

    Seo

    Can you know pest control?????

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.