Robots

Robots is a file that allows search engines to find all the pages on a website.