1# robots.txt for http://www.wikipedia.org/ and friends 2# 3# Please note: There are a lot of pages on this site, and there are 4# some misbehaved spiders out there that go _way_ too fast. If you're 5# irresponsible, your access to the site may be blocked. 6# 7 8# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN 9# and ignoring 429 ratelimit responses, claims to respect robots: 10# http://mj12bot.com/ 11User-agent: MJ12bot 12Disallow: / 13 14# advertising-related bots: 15User-agent: Mediapartners-Google* 16Disallow: / 17 18# Wikipedia work bots: 19User-agent: IsraBot 20Disallow: 21 22User-agent: Orthogaffe 23Disallow: 24 25# Crawlers that are kind enough to obey, but which we'd rather not have 26# unless they're feeding search engines. 27User-agent: UbiCrawler 28Disallow: / 29 30User-agent: DOC 31Disallow: / 32 33User-agent: Zao 34Disallow: / 35 36# Some bots are known to be trouble, particularly those designed to copy 37# entire sites. Please obey robots.txt. 38User-agent: sitecheck.internetseer.com 39Disallow: / 40 41User-agent: Zealbot 42Disallow: / 43 44User-agent: MSIECrawler 45Disallow: / 46 47User-agent: SiteSnagger 48Disallow: / 49 50User-agent: WebStripper 51Disallow: / 52 53User-agent: WebCopier 54Disallow: / 55 56User-agent: Fetch 57Disallow: / 58 59User-agent: Offline Explorer 60Disallow: / 61 62User-agent: Teleport 63Disallow: / 64 65User-agent: TeleportPro 66Disallow: / 67 68User-agent: WebZIP 69Disallow: / 70 71User-agent: linko 72Disallow: / 73 74User-agent: HTTrack 75Disallow: / 76 77User-agent: Microsoft.URL.Control 78Disallow: / 79 80User-agent: Xenu 81Disallow: / 82 83User-agent: larbin 84Disallow: / 85 86User-agent: libwww 87Disallow: / 88 89User-agent: ZyBORG 90Disallow: / 91 92User-agent: Download Ninja 93Disallow: / 94 95# Misbehaving: requests much too fast: 96User-agent: fast 97Disallow: / 98 99# 100# Sorry, wget in its recursive mode is a frequent problem. 101# Please read the man page and use it properly; there is a 102# --wait option you can use to set the delay between hits, 103# for instance. 104# 105User-agent: wget 106Disallow: / 107 108# 109# The 'grub' distributed client has been *very* poorly behaved. 110# 111User-agent: grub-client 112Disallow: / 113 114# 115# Doesn't follow robots.txt anyway, but... 116# 117User-agent: k2spider 118Disallow: / 119 120# 121# Hits many times per second, not acceptable 122# http://www.nameprotect.com/botinfo.html 123User-agent: NPBot 124Disallow: / 125 126# A capture bot, downloads gazillions of pages with no public benefit 127# http://www.webreaper.net/ 128User-agent: WebReaper 129Disallow: / 130 131# Wayback Machine: defaults and whether to index user-pages 132# FIXME: Complete the removal of this block, per T7582. 133# User-agent: archive.org_bot 134# Allow: / 135 136 137# 138# Friendly, low-speed bots are welcome viewing article pages, but not 139# dynamically-generated pages please. 140# 141# Inktomi's "Slurp" can read a minimum delay between hits; if your 142# bot supports such a thing using the 'Crawl-delay' or another 143# instruction, please let us know. 144# 145# There is a special exception for API mobileview to allow dynamic 146# mobile web & app views to load section content. 147# These views aren't HTTP-cached but use parser cache aggressively 148# and don't expose special: pages etc. 149# 150# Another exception is for REST API documentation, located at 151# /api/rest_v1/?doc. 152# 153User-agent: * 154Allow: /w/api.php?action=mobileview& 155Allow: /w/load.php? 156Allow: /api/rest_v1/?doc 157Disallow: /w/ 158Disallow: /api/ 159Disallow: /trap/ 160Disallow: /wiki/Special: 161Disallow: /wiki/Spezial: 162Disallow: /wiki/Spesial: 163Disallow: /wiki/Special%3A 164Disallow: /wiki/Spezial%3A 165Disallow: /wiki/Spesial%3A 166 167# 168# ar: 169Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search 170Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch 171# 172# dewiki: 173# T6937 174# sensible deletion and meta user discussion pages: 175Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/ 176Disallow: /wiki/Wikipedia:Löschkandidaten/ 177Disallow: /wiki/Wikipedia:Vandalensperrung/ 178Disallow: /wiki/Wikipedia:Benutzersperrung/ 179Disallow: /wiki/Wikipedia:Vermittlungsausschuss/ 180Disallow: /wiki/Wikipedia:Administratoren/Probleme/ 181Disallow: /wiki/Wikipedia:Adminkandidaturen/ 182Disallow: /wiki/Wikipedia:Qualitätssicherung/ 183Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/ 184# 4937#5 185Disallow: /wiki/Wikipedia:Vandalismusmeldung/ 186Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/ 187Disallow: /wiki/Wikipedia:Löschprüfung/ 188Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/ 189Disallow: /wiki/Wikipedia:Administratoren/Notizen/ 190Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/ 191Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/ 192# T14111 193Disallow: /wiki/Wikipedia:Checkuser/ 194Disallow: /wiki/Wikipedia_Diskussion:Checkuser/ 195Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/ 196# T15961 197Disallow: /wiki/Wikipedia:Spam-Blacklist-Log 198Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log 199Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log 200Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log 201# 202# enwiki: 203# Folks get annoyed when VfD discussions end up the number 1 google hit for 204# their name. See T6776 205Disallow: /wiki/Wikipedia:Articles_for_deletion/ 206Disallow: /wiki/Wikipedia%3AArticles_for_deletion/ 207Disallow: /wiki/Wikipedia:Votes_for_deletion/ 208Disallow: /wiki/Wikipedia%3AVotes_for_deletion/ 209Disallow: /wiki/Wikipedia:Pages_for_deletion/ 210Disallow: /wiki/Wikipedia%3APages_for_deletion/ 211Disallow: /wiki/Wikipedia:Miscellany_for_deletion/ 212Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/ 213Disallow: /wiki/Wikipedia:Miscellaneous_deletion/ 214Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/ 215Disallow: /wiki/Wikipedia:Copyright_problems 216Disallow: /wiki/Wikipedia%3ACopyright_problems 217Disallow: /wiki/Wikipedia:Protected_titles/ 218Disallow: /wiki/Wikipedia%3AProtected_titles/ 219# T15398 220Disallow: /wiki/Wikipedia:WikiProject_Spam/ 221Disallow: /wiki/Wikipedia%3AWikiProject_Spam/ 222# T16075 223Disallow: /wiki/MediaWiki:Spam-blacklist 224Disallow: /wiki/MediaWiki%3ASpam-blacklist 225Disallow: /wiki/MediaWiki_talk:Spam-blacklist 226Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist 227# T13261 228Disallow: /wiki/Wikipedia:Requests_for_arbitration/ 229Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/ 230Disallow: /wiki/Wikipedia:Requests_for_comment/ 231Disallow: /wiki/Wikipedia%3ARequests_for_comment/ 232Disallow: /wiki/Wikipedia:Requests_for_adminship/ 233Disallow: /wiki/Wikipedia%3ARequests_for_adminship/ 234# T12288 235Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/ 236Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/ 237Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/ 238Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/ 239Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/ 240Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/ 241Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/ 242Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/ 243Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/ 244Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/ 245# T16793 246Disallow: /wiki/Wikipedia:Changing_username 247Disallow: /wiki/Wikipedia%3AChanging_username 248Disallow: /wiki/Wikipedia:Changing_username/ 249Disallow: /wiki/Wikipedia%3AChanging_username/ 250Disallow: /wiki/Wikipedia_talk:Changing_username 251Disallow: /wiki/Wikipedia_talk%3AChanging_username 252Disallow: /wiki/Wikipedia_talk:Changing_username/ 253Disallow: /wiki/Wikipedia_talk%3AChanging_username/ 254# 255# eswiki: 256# T8746 257Disallow: /wiki/Wikipedia:Consultas_de_borrado/ 258Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/ 259# 260# fiwiki: 261# T10695 262Disallow: /wiki/Wikipedia:Poistettavat_sivut 263Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4: 264Disallow: /wiki/Käyttäjä: 265Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4: 266Disallow: /wiki/Keskustelu_käyttäjästä: 267Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/ 268Disallow: /wiki/Wikipedia:Ylläpitäjät/ 269# 270# hewiki: 271Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search 272Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch 273#T11517 274Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/ 275Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/ 276Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/ 277Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/ 278Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים 279Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים 280Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D 281Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D 282Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים 283Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים 284Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D 285Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D 286# 287# huwiki: 288Disallow: /wiki/Speci%C3%A1lis:Search 289Disallow: /wiki/Speci%C3%A1lis%3ASearch 290# 291# itwiki: 292# T7545 293Disallow: /wiki/Wikipedia:Pagine_da_cancellare 294Disallow: /wiki/Wikipedia%3APagine_da_cancellare 295Disallow: /wiki/Wikipedia:Utenti_problematici 296Disallow: /wiki/Wikipedia%3AUtenti_problematici 297Disallow: /wiki/Wikipedia:Vandalismi_in_corso 298Disallow: /wiki/Wikipedia%3AVandalismi_in_corso 299Disallow: /wiki/Wikipedia:Amministratori 300Disallow: /wiki/Wikipedia%3AAmministratori 301Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata 302Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata 303Disallow: /wiki/Categoria:Da_cancellare_subito 304Disallow: /wiki/Categoria%3ADa_cancellare_subito 305Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright 306Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright 307Disallow: /wiki/Categoria:Da_controllare_per_copyright 308Disallow: /wiki/Categoria%3ADa_controllare_per_copyright 309Disallow: /wiki/Progetto:Rimozione_contributi_sospetti 310Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti 311Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright 312Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright 313Disallow: /wiki/Progetto:Cococo 314Disallow: /wiki/Progetto%3ACococo 315Disallow: /wiki/Discussioni_progetto:Cococo 316Disallow: /wiki/Discussioni_progetto%3ACococo 317# 318# jawiki 319Disallow: /wiki/%E7%89%B9%E5%88%A5:Search 320Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch 321# T7239 322Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/ 323Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/ 324Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC 325Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC 326# nowiki 327# T13432 328Disallow: /wiki/Bruker: 329Disallow: /wiki/Bruker%3A 330Disallow: /wiki/Brukerdiskusjon 331Disallow: /wiki/Wikipedia:Administratorer 332Disallow: /wiki/Wikipedia%3AAdministratorer 333Disallow: /wiki/Wikipedia-diskusjon:Administratorer 334Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer 335Disallow: /wiki/Wikipedia:Sletting 336Disallow: /wiki/Wikipedia%3ASletting 337Disallow: /wiki/Wikipedia-diskusjon:Sletting 338Disallow: /wiki/Wikipedia-diskusjon%3ASletting 339# 340# plwiki 341# T10067 342Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia 343Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia 344Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia 345Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia 346Disallow: /wiki/Wikipedia:SDU/ 347Disallow: /wiki/Wikipedia%3ASDU/ 348Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich 349Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich 350# 351# ptwiki: 352# T7394 353Disallow: /wiki/Wikipedia:Páginas_para_eliminar/ 354Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/ 355Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/ 356Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/ 357Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/ 358Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/ 359# 360# rowiki: 361# T14546 362Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters 363Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters 364Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters 365Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters 366# 367# ruwiki: 368Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search 369Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch 370# 371# svwiki: 372# T12229 373Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering 374Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering 375Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering 376Disallow: /wiki/Användare 377Disallow: /wiki/Anv%C3%A4ndare 378Disallow: /wiki/Användardiskussion 379Disallow: /wiki/Anv%C3%A4ndardiskussion 380Disallow: /wiki/Wikipedia:Skyddade_sidnamn 381Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn 382# T13291 383Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas 384Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas 385Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas 386# 387# zhwiki: 388# T7104 389Disallow: /wiki/Wikipedia:删除投票/侵权 390Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83 391Disallow: /wiki/Wikipedia:删除投票和请求 392Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82 393Disallow: /wiki/Category:快速删除候选 394Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89 395Disallow: /wiki/Category:维基百科需要翻译的文章 396Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0 397# 398# sister projects 399# 400# enwikinews: 401# T7340 402Disallow: /wiki/Portal:Prepared_stories/ 403Disallow: /wiki/Portal%3APrepared_stories/ 404# 405# itwikinews 406# T11138 407Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione 408Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright 409Disallow: /wiki/Categoria:Da_cancellare_subito 410Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright 411Disallow: /wiki/Wikinotizie:Storie_in_preparazione 412# 413# enwikiquote: 414# T17095 415Disallow: /wiki/Wikiquote:Votes_for_deletion/ 416Disallow: /wiki/Wikiquote%3AVotes_for_deletion/ 417Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/ 418Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/ 419Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/ 420Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/ 421Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/ 422Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/ 423# 424# enwikibooks 425Disallow: /wiki/Wikibooks:Votes_for_deletion 426# 427# working... 428Disallow: /wiki/Fundraising_2007/comments 429# 430# 431# 432#----------------------------------------------------------# 433# 434# 435# 436