1:mod:`ast` --- Abstract Syntax Trees 2==================================== 3 4.. module:: ast 5 :synopsis: Abstract Syntax Tree classes and manipulation. 6 7.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> 8.. sectionauthor:: Georg Brandl <georg@python.org> 9 10**Source code:** :source:`Lib/ast.py` 11 12-------------- 13 14The :mod:`ast` module helps Python applications to process trees of the Python 15abstract syntax grammar. The abstract syntax itself might change with each 16Python release; this module helps to find out programmatically what the current 17grammar looks like. 18 19An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as 20a flag to the :func:`compile` built-in function, or using the :func:`parse` 21helper provided in this module. The result will be a tree of objects whose 22classes all inherit from :class:`ast.AST`. An abstract syntax tree can be 23compiled into a Python code object using the built-in :func:`compile` function. 24 25 26Node classes 27------------ 28 29.. class:: AST 30 31 This is the base of all AST node classes. The actual node classes are 32 derived from the :file:`Parser/Python.asdl` file, which is reproduced 33 :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C 34 module and re-exported in :mod:`ast`. 35 36 There is one class defined for each left-hand side symbol in the abstract 37 grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, 38 there is one class defined for each constructor on the right-hand side; these 39 classes inherit from the classes for the left-hand side trees. For example, 40 :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules 41 with alternatives (aka "sums"), the left-hand side class is abstract: only 42 instances of specific constructor nodes are ever created. 43 44 .. index:: single: ? (question mark); in AST grammar 45 .. index:: single: * (asterisk); in AST grammar 46 47 .. attribute:: _fields 48 49 Each concrete class has an attribute :attr:`_fields` which gives the names 50 of all child nodes. 51 52 Each instance of a concrete class has one attribute for each child node, 53 of the type as defined in the grammar. For example, :class:`ast.BinOp` 54 instances have an attribute :attr:`left` of type :class:`ast.expr`. 55 56 If these attributes are marked as optional in the grammar (using a 57 question mark), the value might be ``None``. If the attributes can have 58 zero-or-more values (marked with an asterisk), the values are represented 59 as Python lists. All possible attributes must be present and have valid 60 values when compiling an AST with :func:`compile`. 61 62 .. attribute:: lineno 63 col_offset 64 end_lineno 65 end_col_offset 66 67 Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have 68 :attr:`lineno`, :attr:`col_offset`, :attr:`lineno`, and :attr:`col_offset` 69 attributes. The :attr:`lineno` and :attr:`end_lineno` are the first and 70 last line numbers of source text span (1-indexed so the first line is line 1) 71 and the :attr:`col_offset` and :attr:`end_col_offset` are the corresponding 72 UTF-8 byte offsets of the first and last tokens that generated the node. 73 The UTF-8 offset is recorded because the parser uses UTF-8 internally. 74 75 Note that the end positions are not required by the compiler and are 76 therefore optional. The end offset is *after* the last symbol, for example 77 one can get the source segment of a one-line expression node using 78 ``source_line[node.col_offset : node.end_col_offset]``. 79 80 The constructor of a class :class:`ast.T` parses its arguments as follows: 81 82 * If there are positional arguments, there must be as many as there are items 83 in :attr:`T._fields`; they will be assigned as attributes of these names. 84 * If there are keyword arguments, they will set the attributes of the same 85 names to the given values. 86 87 For example, to create and populate an :class:`ast.UnaryOp` node, you could 88 use :: 89 90 node = ast.UnaryOp() 91 node.op = ast.USub() 92 node.operand = ast.Constant() 93 node.operand.value = 5 94 node.operand.lineno = 0 95 node.operand.col_offset = 0 96 node.lineno = 0 97 node.col_offset = 0 98 99 or the more compact :: 100 101 node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0), 102 lineno=0, col_offset=0) 103 104.. versionchanged:: 3.8 105 106 Class :class:`ast.Constant` is now used for all constants. 107 108.. deprecated:: 3.8 109 110 Old classes :class:`ast.Num`, :class:`ast.Str`, :class:`ast.Bytes`, 111 :class:`ast.NameConstant` and :class:`ast.Ellipsis` are still available, 112 but they will be removed in future Python releases. In the meanwhile, 113 instantiating them will return an instance of a different class. 114 115 116.. _abstract-grammar: 117 118Abstract Grammar 119---------------- 120 121The abstract grammar is currently defined as follows: 122 123.. literalinclude:: ../../Parser/Python.asdl 124 :language: none 125 126 127:mod:`ast` Helpers 128------------------ 129 130Apart from the node classes, the :mod:`ast` module defines these utility functions 131and classes for traversing abstract syntax trees: 132 133.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False, feature_version=None) 134 135 Parse the source into an AST node. Equivalent to ``compile(source, 136 filename, mode, ast.PyCF_ONLY_AST)``. 137 138 If ``type_comments=True`` is given, the parser is modified to check 139 and return type comments as specified by :pep:`484` and :pep:`526`. 140 This is equivalent to adding :data:`ast.PyCF_TYPE_COMMENTS` to the 141 flags passed to :func:`compile()`. This will report syntax errors 142 for misplaced type comments. Without this flag, type comments will 143 be ignored, and the ``type_comment`` field on selected AST nodes 144 will always be ``None``. In addition, the locations of ``# type: 145 ignore`` comments will be returned as the ``type_ignores`` 146 attribute of :class:`Module` (otherwise it is always an empty list). 147 148 In addition, if ``mode`` is ``'func_type'``, the input syntax is 149 modified to correspond to :pep:`484` "signature type comments", 150 e.g. ``(str, int) -> List[str]``. 151 152 Also, setting ``feature_version`` to a tuple ``(major, minor)`` 153 will attempt to parse using that Python version's grammar. 154 Currently ``major`` must equal to ``3``. For example, setting 155 ``feature_version=(3, 4)`` will allow the use of ``async`` and 156 ``await`` as variable names. The lowest supported version is 157 ``(3, 4)``; the highest is ``sys.version_info[0:2]``. 158 159 .. warning:: 160 It is possible to crash the Python interpreter with a 161 sufficiently large/complex string due to stack depth limitations 162 in Python's AST compiler. 163 164 .. versionchanged:: 3.8 165 Added ``type_comments``, ``mode='func_type'`` and ``feature_version``. 166 167 168.. function:: literal_eval(node_or_string) 169 170 Safely evaluate an expression node or a string containing a Python literal or 171 container display. The string or node provided may only consist of the 172 following Python literal structures: strings, bytes, numbers, tuples, lists, 173 dicts, sets, booleans, and ``None``. 174 175 This can be used for safely evaluating strings containing Python values from 176 untrusted sources without the need to parse the values oneself. It is not 177 capable of evaluating arbitrarily complex expressions, for example involving 178 operators or indexing. 179 180 .. warning:: 181 It is possible to crash the Python interpreter with a 182 sufficiently large/complex string due to stack depth limitations 183 in Python's AST compiler. 184 185 .. versionchanged:: 3.2 186 Now allows bytes and set literals. 187 188 189.. function:: get_docstring(node, clean=True) 190 191 Return the docstring of the given *node* (which must be a 192 :class:`FunctionDef`, :class:`AsyncFunctionDef`, :class:`ClassDef`, 193 or :class:`Module` node), or ``None`` if it has no docstring. 194 If *clean* is true, clean up the docstring's indentation with 195 :func:`inspect.cleandoc`. 196 197 .. versionchanged:: 3.5 198 :class:`AsyncFunctionDef` is now supported. 199 200 201.. function:: get_source_segment(source, node, *, padded=False) 202 203 Get source code segment of the *source* that generated *node*. 204 If some location information (:attr:`lineno`, :attr:`end_lineno`, 205 :attr:`col_offset`, or :attr:`end_col_offset`) is missing, return ``None``. 206 207 If *padded* is ``True``, the first line of a multi-line statement will 208 be padded with spaces to match its original position. 209 210 .. versionadded:: 3.8 211 212 213.. function:: fix_missing_locations(node) 214 215 When you compile a node tree with :func:`compile`, the compiler expects 216 :attr:`lineno` and :attr:`col_offset` attributes for every node that supports 217 them. This is rather tedious to fill in for generated nodes, so this helper 218 adds these attributes recursively where not already set, by setting them to 219 the values of the parent node. It works recursively starting at *node*. 220 221 222.. function:: increment_lineno(node, n=1) 223 224 Increment the line number and end line number of each node in the tree 225 starting at *node* by *n*. This is useful to "move code" to a different 226 location in a file. 227 228 229.. function:: copy_location(new_node, old_node) 230 231 Copy source location (:attr:`lineno`, :attr:`col_offset`, :attr:`end_lineno`, 232 and :attr:`end_col_offset`) from *old_node* to *new_node* if possible, 233 and return *new_node*. 234 235 236.. function:: iter_fields(node) 237 238 Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` 239 that is present on *node*. 240 241 242.. function:: iter_child_nodes(node) 243 244 Yield all direct child nodes of *node*, that is, all fields that are nodes 245 and all items of fields that are lists of nodes. 246 247 248.. function:: walk(node) 249 250 Recursively yield all descendant nodes in the tree starting at *node* 251 (including *node* itself), in no specified order. This is useful if you only 252 want to modify nodes in place and don't care about the context. 253 254 255.. class:: NodeVisitor() 256 257 A node visitor base class that walks the abstract syntax tree and calls a 258 visitor function for every node found. This function may return a value 259 which is forwarded by the :meth:`visit` method. 260 261 This class is meant to be subclassed, with the subclass adding visitor 262 methods. 263 264 .. method:: visit(node) 265 266 Visit a node. The default implementation calls the method called 267 :samp:`self.visit_{classname}` where *classname* is the name of the node 268 class, or :meth:`generic_visit` if that method doesn't exist. 269 270 .. method:: generic_visit(node) 271 272 This visitor calls :meth:`visit` on all children of the node. 273 274 Note that child nodes of nodes that have a custom visitor method won't be 275 visited unless the visitor calls :meth:`generic_visit` or visits them 276 itself. 277 278 Don't use the :class:`NodeVisitor` if you want to apply changes to nodes 279 during traversal. For this a special visitor exists 280 (:class:`NodeTransformer`) that allows modifications. 281 282 .. deprecated:: 3.8 283 284 Methods :meth:`visit_Num`, :meth:`visit_Str`, :meth:`visit_Bytes`, 285 :meth:`visit_NameConstant` and :meth:`visit_Ellipsis` are deprecated 286 now and will not be called in future Python versions. Add the 287 :meth:`visit_Constant` method to handle all constant nodes. 288 289 290.. class:: NodeTransformer() 291 292 A :class:`NodeVisitor` subclass that walks the abstract syntax tree and 293 allows modification of nodes. 294 295 The :class:`NodeTransformer` will walk the AST and use the return value of 296 the visitor methods to replace or remove the old node. If the return value 297 of the visitor method is ``None``, the node will be removed from its 298 location, otherwise it is replaced with the return value. The return value 299 may be the original node in which case no replacement takes place. 300 301 Here is an example transformer that rewrites all occurrences of name lookups 302 (``foo``) to ``data['foo']``:: 303 304 class RewriteName(NodeTransformer): 305 306 def visit_Name(self, node): 307 return Subscript( 308 value=Name(id='data', ctx=Load()), 309 slice=Index(value=Constant(value=node.id)), 310 ctx=node.ctx 311 ) 312 313 Keep in mind that if the node you're operating on has child nodes you must 314 either transform the child nodes yourself or call the :meth:`generic_visit` 315 method for the node first. 316 317 For nodes that were part of a collection of statements (that applies to all 318 statement nodes), the visitor may also return a list of nodes rather than 319 just a single node. 320 321 If :class:`NodeTransformer` introduces new nodes (that weren't part of 322 original tree) without giving them location information (such as 323 :attr:`lineno`), :func:`fix_missing_locations` should be called with 324 the new sub-tree to recalculate the location information:: 325 326 tree = ast.parse('foo', mode='eval') 327 new_tree = fix_missing_locations(RewriteName().visit(tree)) 328 329 Usually you use the transformer like this:: 330 331 node = YourTransformer().visit(node) 332 333 334.. function:: dump(node, annotate_fields=True, include_attributes=False) 335 336 Return a formatted dump of the tree in *node*. This is mainly useful for 337 debugging purposes. If *annotate_fields* is true (by default), 338 the returned string will show the names and the values for fields. 339 If *annotate_fields* is false, the result string will be more compact by 340 omitting unambiguous field names. Attributes such as line 341 numbers and column offsets are not dumped by default. If this is wanted, 342 *include_attributes* can be set to true. 343 344.. seealso:: 345 346 `Green Tree Snakes <https://greentreesnakes.readthedocs.io/>`_, an external 347 documentation resource, has good details on working with Python ASTs. 348 349 `ASTTokens <https://asttokens.readthedocs.io/en/latest/user-guide.html>`_ 350 annotates Python ASTs with the positions of tokens and text in the source 351 code that generated them. This is helpful for tools that make source code 352 transformations. 353 354 `leoAst.py <http://leoeditor.com/appendices.html#leoast-py>`_ unifies the 355 token-based and parse-tree-based views of python programs by inserting 356 two-way links between tokens and ast nodes. 357 358 `LibCST <https://libcst.readthedocs.io/>`_ parses code as a Concrete Syntax 359 Tree that looks like an ast tree and keeps all formatting details. It's 360 useful for building automated refactoring (codemod) applications and 361 linters. 362 363 `Parso <https://parso.readthedocs.io>`_ is a Python parser that supports 364 error recovery and round-trip parsing for different Python versions (in 365 multiple Python versions). Parso is also able to list multiple syntax errors 366 in your python file.