1User-agent: * 2Disallow: /*/print$ 3# Don't allow indexing of user needs pages 4Disallow: /info/* 5Sitemap: https://www.gov.uk/sitemap.xml 6 7# https://ahrefs.com/robot/ crawls the site frequently 8User-agent: AhrefsBot 9Crawl-delay: 10 10 11# https://www.deepcrawl.com/bot/ makes lots of requests. Ideally 12# we'd slow it down rather than blocking it but it doesn't mention 13# whether or not it supports crawl-delay. 14User-agent: deepcrawl 15Disallow: / 16 17# Complaints of 429 'Too many requests' seem to be coming from SharePoint servers 18# (https://social.msdn.microsoft.com/Forums/en-US/3ea268ed-58a6-4166-ab40-d3f4fc55fef4) 19# The robot doesn't recognise its User-Agent string, see the MS support article: 20# https://support.microsoft.com/en-us/help/3019711/the-sharepoint-server-crawler-ignores-directives-in-robots-txt 21User-agent: MS Search 6.0 Robot 22Disallow: / 23