robots.txt tells it what to do Spiders crawl pages and follow links Quality Raters check things manually