1# Robots.txt file for https://9gag.com 2# All robots will spider the domain 3 4User-agent: Mediapartners-Google 5Disallow: 6 7User-agent: * 8Disallow: /u/ 9Disallow: /read/ 10Disallow: /gag-in-app/* 11Disallow: /views?* 12Disallow: /search* 13Disallow: /search?query=* 14Disallow: /pref/* 15Disallow: /settings* 16Disallow: /notifications* 17Disallow: /views* 18Disallow: /connect/* 19Disallow: /others/* 20Disallow: /static/* 21Disallow: /static/ads/* 22Disallow: /l.php 23Disallow: /? 24 25Crawl-delay: 1 26 27