1Sitemap: https://www.fbi.gov/sitemap.xml.gz 2 3# Define access-restrictions for robots/spiders 4# http://www.robotstxt.org/wc/norobots.html 5 6 7 8# By default we allow robots to access all areas of our site 9# already accessible to anonymous users 10 11User-agent: * 12Disallow: 13 14 15 16# Add Googlebot-specific syntax extension to exclude forms 17# that are repeated for each piece of content in the site 18# the wildcard is only supported by Googlebot 19# http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling 20 21User-Agent: Googlebot 22User-Agent: Bingbot 23User-Agent: Slurp 24User-Agent: DuckDuckBot 25 26Disallow: /@@search? 27Disallow: /search? 28Disallow: /*atct_album_view$ 29Disallow: /*folder_factories$ 30Disallow: /*folder_summary_view$ 31Disallow: /*login_form$ 32Disallow: /*mail_password_form$ 33Disallow: /*search_rss$ 34Disallow: /*sendto_form$ 35Disallow: /*summary_view$ 36Disallow: /*thumbnail_view$ 37Disallow: /plonejsi18n$ 38Disallow: /*@@castle.cms.querylisting*?