A robots.txt file allows search engines like google crawl your website more proficiently. It does this by steering the crawlers far from cumbersome media or useful resource data files that aren’t appropriate to Search engine optimisation.You should also be sure your hyperlinks perform. Broken one-way links point out that you’re ignoring correct