User-agent: * Allow: / Disallow: *&id16=* # seconds to wait between requests Crawl-delay: 5 # maximum rate is one page every 5 seconds Request-rate: 1/5 # only visit between 00:00 and 03:00 EST (value below is GMT) Visit-time: 0600-0900