Blog Archives for tag crawlling

What is Robots.txt?

Robots.txt is a simple text file that sits on the server with your web site. It’s basically your web site’s way of giving instructions to search engines about what how they index your web site. Search Engines tend to look for the robots.txt file when they first visit a site. They can visit and index your site whether you have a robots.txt file or not; having one simply helps them… Continue reading »

Last updated by at .