Difference Between Meta robots tag And Robots.txt

META ROBOTS VS ROBOTS.TXT Robots.txt files are best for disallowing a whole section of a site , such as a category whereas a meta tag is more efficient at disallowing single files and pages . Meta Robots tags Meta robots are meta directives . It provides crawlers instructions for how to crawl or index web page content . Meta robots tags can block a single page . Ex . < meta name = " robots " content = " noindex , nofollow " > 10 O Robots.txt Robots.txt is a text file . It gives bots suggestions for how to crawl a website's page . Robot.txt file can block the whole website . Ex . user - agent : Allow : Disallow : / * java T point