User-Agent: * Disallow: User-agent: Yandex Crawl-delay: 2 # specifies a 2 second timeout User-Agent: MJ12bot Crawl-Delay: 5