1mwparserfromhell 2================ 3 4.. image:: https://api.travis-ci.com/earwig/mwparserfromhell.svg?branch=develop 5 :alt: Build Status 6 :target: https://travis-ci.org/earwig/mwparserfromhell 7 8.. image:: https://img.shields.io/coveralls/earwig/mwparserfromhell/develop.svg 9 :alt: Coverage Status 10 :target: https://coveralls.io/r/earwig/mwparserfromhell 11 12**mwparserfromhell** (the *MediaWiki Parser from Hell*) is a Python package 13that provides an easy-to-use and outrageously powerful parser for MediaWiki_ 14wikicode. It supports Python 3.5+. 15 16Developed by Earwig_ with contributions from `Σ`_, Legoktm_, and others. 17Full documentation is available on ReadTheDocs_. Development occurs on GitHub_. 18 19Installation 20------------ 21 22The easiest way to install the parser is through the `Python Package Index`_; 23you can install the latest release with ``pip install mwparserfromhell`` 24(`get pip`_). Make sure your pip is up-to-date first, especially on Windows. 25 26Alternatively, get the latest development version:: 27 28 git clone https://github.com/earwig/mwparserfromhell.git 29 cd mwparserfromhell 30 python setup.py install 31 32The comprehensive unit testing suite requires `pytest`_ (``pip install pytest``) 33and can be run with ``python -m pytest``. 34 35Usage 36----- 37 38Normal usage is rather straightforward (where ``text`` is page text): 39 40>>> import mwparserfromhell 41>>> wikicode = mwparserfromhell.parse(text) 42 43``wikicode`` is a ``mwparserfromhell.Wikicode`` object, which acts like an 44ordinary ``str`` object with some extra methods. For example: 45 46>>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?" 47>>> wikicode = mwparserfromhell.parse(text) 48>>> print(wikicode) 49I has a template! {{foo|bar|baz|eggs=spam}} See it? 50>>> templates = wikicode.filter_templates() 51>>> print(templates) 52['{{foo|bar|baz|eggs=spam}}'] 53>>> template = templates[0] 54>>> print(template.name) 55foo 56>>> print(template.params) 57['bar', 'baz', 'eggs=spam'] 58>>> print(template.get(1).value) 59bar 60>>> print(template.get("eggs").value) 61spam 62 63Since nodes can contain other nodes, getting nested templates is trivial: 64 65>>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}" 66>>> mwparserfromhell.parse(text).filter_templates() 67['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}'] 68 69You can also pass ``recursive=False`` to ``filter_templates()`` and explore 70templates manually. This is possible because nodes can contain additional 71``Wikicode`` objects: 72 73>>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}") 74>>> print(code.filter_templates(recursive=False)) 75['{{foo|this {{includes a|template}}}}'] 76>>> foo = code.filter_templates(recursive=False)[0] 77>>> print(foo.get(1).value) 78this {{includes a|template}} 79>>> print(foo.get(1).value.filter_templates()[0]) 80{{includes a|template}} 81>>> print(foo.get(1).value.filter_templates()[0].get(1).value) 82template 83 84Templates can be easily modified to add, remove, or alter params. ``Wikicode`` 85objects can be treated like lists, with ``append()``, ``insert()``, 86``remove()``, ``replace()``, and more. They also have a ``matches()`` method 87for comparing page or template names, which takes care of capitalization and 88whitespace: 89 90>>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}" 91>>> code = mwparserfromhell.parse(text) 92>>> for template in code.filter_templates(): 93... if template.name.matches("Cleanup") and not template.has("date"): 94... template.add("date", "July 2012") 95... 96>>> print(code) 97{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}} 98>>> code.replace("{{uncategorized}}", "{{bar-stub}}") 99>>> print(code) 100{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}} 101>>> print(code.filter_templates()) 102['{{cleanup|date=July 2012}}', '{{bar-stub}}'] 103 104You can then convert ``code`` back into a regular ``str`` object (for 105saving the page!) by calling ``str()`` on it: 106 107>>> text = str(code) 108>>> print(text) 109{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}} 110>>> text == code 111True 112 113Limitations 114----------- 115 116While the MediaWiki parser generates HTML and has access to the contents of 117templates, among other things, mwparserfromhell acts as a direct interface to 118the source code only. This has several implications: 119 120* Syntax elements produced by a template transclusion cannot be detected. For 121 example, imagine a hypothetical page ``"Template:End-bold"`` that contained 122 the text ``</b>``. While MediaWiki would correctly understand that 123 ``<b>foobar{{end-bold}}`` translates to ``<b>foobar</b>``, mwparserfromhell 124 has no way of examining the contents of ``{{end-bold}}``. Instead, it would 125 treat the bold tag as unfinished, possibly extending further down the page. 126 127* Templates adjacent to external links, as in ``http://example.com{{foo}}``, 128 are considered part of the link. In reality, this would depend on the 129 contents of the template. 130 131* When different syntax elements cross over each other, as in 132 ``{{echo|''Hello}}, world!''``, the parser gets confused because this cannot 133 be represented by an ordinary syntax tree. Instead, the parser will treat the 134 first syntax construct as plain text. In this case, only the italic tag would 135 be properly parsed. 136 137 **Workaround:** Since this commonly occurs with text formatting and text 138 formatting is often not of interest to users, you may pass 139 *skip_style_tags=True* to ``mwparserfromhell.parse()``. This treats ``''`` 140 and ``'''`` as plain text. 141 142 A future version of mwparserfromhell may include multiple parsing modes to 143 get around this restriction more sensibly. 144 145Additionally, the parser lacks awareness of certain wiki-specific settings: 146 147* `Word-ending links`_ are not supported, since the linktrail rules are 148 language-specific. 149 150* Localized namespace names aren't recognized, so file links (such as 151 ``[[File:...]]``) are treated as regular wikilinks. 152 153* Anything that looks like an XML tag is treated as a tag, even if it is not a 154 recognized tag name, since the list of valid tags depends on loaded MediaWiki 155 extensions. 156 157Integration 158----------- 159 160``mwparserfromhell`` is used by and originally developed for EarwigBot_; 161``Page`` objects have a ``parse`` method that essentially calls 162``mwparserfromhell.parse()`` on ``page.get()``. 163 164If you're using Pywikibot_, your code might look like this: 165 166.. code-block:: python 167 168 import mwparserfromhell 169 import pywikibot 170 171 def parse(title): 172 site = pywikibot.Site() 173 page = pywikibot.Page(site, title) 174 text = page.get() 175 return mwparserfromhell.parse(text) 176 177If you're not using a library, you can parse any page with the following 178Python 3 code (using the API_ and the requests_ library): 179 180.. code-block:: python 181 182 import mwparserfromhell 183 import requests 184 185 API_URL = "https://en.wikipedia.org/w/api.php" 186 187 def parse(title): 188 params = { 189 "action": "query", 190 "prop": "revisions", 191 "rvprop": "content", 192 "rvslots": "main", 193 "rvlimit": 1, 194 "titles": title, 195 "format": "json", 196 "formatversion": "2", 197 } 198 headers = {"User-Agent": "My-Bot-Name/1.0"} 199 req = requests.get(API_URL, headers=headers, params=params) 200 res = req.json() 201 revision = res["query"]["pages"][0]["revisions"][0] 202 text = revision["slots"]["main"]["content"] 203 return mwparserfromhell.parse(text) 204 205.. _MediaWiki: https://www.mediawiki.org 206.. _ReadTheDocs: https://mwparserfromhell.readthedocs.io 207.. _Earwig: https://en.wikipedia.org/wiki/User:The_Earwig 208.. _Σ: https://en.wikipedia.org/wiki/User:%CE%A3 209.. _Legoktm: https://en.wikipedia.org/wiki/User:Legoktm 210.. _GitHub: https://github.com/earwig/mwparserfromhell 211.. _Python Package Index: https://pypi.org/ 212.. _get pip: https://pypi.org/project/pip/ 213.. _pytest: https://docs.pytest.org/ 214.. _Word-ending links: https://www.mediawiki.org/wiki/Help:Links#linktrail 215.. _EarwigBot: https://github.com/earwig/earwigbot 216.. _Pywikibot: https://www.mediawiki.org/wiki/Manual:Pywikibot 217.. _API: https://www.mediawiki.org/wiki/API:Main_page 218.. _requests: https://2.python-requests.org 219