1**************************** 2 What's New In Python 3.2 3**************************** 4 5:Author: Raymond Hettinger 6 7.. $Id$ 8 Rules for maintenance: 9 10 * Anyone can add text to this document. Do not spend very much time 11 on the wording of your changes, because your text will probably 12 get rewritten. (Note, during release candidate phase or just before 13 a beta release, please use the tracker instead -- this helps avoid 14 merge conflicts. If you must add a suggested entry directly, 15 please put it in an XXX comment and the maintainer will take notice). 16 17 * The maintainer will go through Misc/NEWS periodically and add 18 changes; it's therefore more important to add your changes to 19 Misc/NEWS than to this file. 20 21 * This is not a complete list of every single change; completeness 22 is the purpose of Misc/NEWS. Some changes I consider too small 23 or esoteric to include. If such a change is added to the text, 24 I'll just remove it. (This is another reason you shouldn't spend 25 too much time on writing your addition.) 26 27 * If you want to draw your new text to the attention of the 28 maintainer, add 'XXX' to the beginning of the paragraph or 29 section. 30 31 * It's OK to just add a fragmentary note about a change. For 32 example: "XXX Describe the transmogrify() function added to the 33 socket module." The maintainer will research the change and 34 write the necessary text. 35 36 * You can comment out your additions if you like, but it's not 37 necessary (especially when a final release is some months away). 38 39 * Credit the author of a patch or bugfix. Just the name is 40 sufficient; the e-mail address isn't necessary. It's helpful to 41 add the issue number: 42 43 XXX Describe the transmogrify() function added to the socket 44 module. 45 46 (Contributed by P.Y. Developer; :issue:`12345`.) 47 48 This saves the maintainer the effort of going through the SVN log 49 when researching a change. 50 51This article explains the new features in Python 3.2 as compared to 3.1. It 52focuses on a few highlights and gives a few examples. For full details, see the 53`Misc/NEWS 54<https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`_ 55file. 56 57.. seealso:: 58 59 :pep:`392` - Python 3.2 Release Schedule 60 61 62PEP 384: Defining a Stable ABI 63============================== 64 65In the past, extension modules built for one Python version were often 66not usable with other Python versions. Particularly on Windows, every 67feature release of Python required rebuilding all extension modules that 68one wanted to use. This requirement was the result of the free access to 69Python interpreter internals that extension modules could use. 70 71With Python 3.2, an alternative approach becomes available: extension 72modules which restrict themselves to a limited API (by defining 73Py_LIMITED_API) cannot use many of the internals, but are constrained 74to a set of API functions that are promised to be stable for several 75releases. As a consequence, extension modules built for 3.2 in that 76mode will also work with 3.3, 3.4, and so on. Extension modules that 77make use of details of memory structures can still be built, but will 78need to be recompiled for every feature release. 79 80.. seealso:: 81 82 :pep:`384` - Defining a Stable ABI 83 PEP written by Martin von Löwis. 84 85 86PEP 389: Argparse Command Line Parsing Module 87============================================= 88 89A new module for command line parsing, :mod:`argparse`, was introduced to 90overcome the limitations of :mod:`optparse` which did not provide support for 91positional arguments (not just options), subcommands, required options and other 92common patterns of specifying and validating options. 93 94This module has already had widespread success in the community as a 95third-party module. Being more fully featured than its predecessor, the 96:mod:`argparse` module is now the preferred module for command-line processing. 97The older module is still being kept available because of the substantial amount 98of legacy code that depends on it. 99 100Here's an annotated example parser showing features like limiting results to a 101set of choices, specifying a *metavar* in the help screen, validating that one 102or more positional arguments is present, and making a required option:: 103 104 import argparse 105 parser = argparse.ArgumentParser( 106 description = 'Manage servers', # main description for help 107 epilog = 'Tested on Solaris and Linux') # displayed after help 108 parser.add_argument('action', # argument name 109 choices = ['deploy', 'start', 'stop'], # three allowed values 110 help = 'action on each target') # help msg 111 parser.add_argument('targets', 112 metavar = 'HOSTNAME', # var name used in help msg 113 nargs = '+', # require one or more targets 114 help = 'url for target machines') # help msg explanation 115 parser.add_argument('-u', '--user', # -u or --user option 116 required = True, # make it a required argument 117 help = 'login as user') 118 119Example of calling the parser on a command string:: 120 121 >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' 122 >>> result = parser.parse_args(cmd.split()) 123 >>> result.action 124 'deploy' 125 >>> result.targets 126 ['sneezy.example.com', 'sleepy.example.com'] 127 >>> result.user 128 'skycaptain' 129 130Example of the parser's automatically generated help:: 131 132 >>> parser.parse_args('-h'.split()) 133 134 usage: manage_cloud.py [-h] -u USER 135 {deploy,start,stop} HOSTNAME [HOSTNAME ...] 136 137 Manage servers 138 139 positional arguments: 140 {deploy,start,stop} action on each target 141 HOSTNAME url for target machines 142 143 optional arguments: 144 -h, --help show this help message and exit 145 -u USER, --user USER login as user 146 147 Tested on Solaris and Linux 148 149An especially nice :mod:`argparse` feature is the ability to define subparsers, 150each with their own argument patterns and help displays:: 151 152 import argparse 153 parser = argparse.ArgumentParser(prog='HELM') 154 subparsers = parser.add_subparsers() 155 156 parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup 157 parser_l.add_argument('-m', '--missiles', action='store_true') 158 parser_l.add_argument('-t', '--torpedos', action='store_true') 159 160 parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup 161 aliases=('steer', 'turn')) # equivalent names 162 parser_m.add_argument('-c', '--course', type=int, required=True) 163 parser_m.add_argument('-s', '--speed', type=int, default=0) 164 165.. code-block:: shell-session 166 167 $ ./helm.py --help # top level help (launch and move) 168 $ ./helm.py launch --help # help for launch options 169 $ ./helm.py launch --missiles # set missiles=True and torpedos=False 170 $ ./helm.py steer --course 180 --speed 5 # set movement parameters 171 172.. seealso:: 173 174 :pep:`389` - New Command Line Parsing Module 175 PEP written by Steven Bethard. 176 177 :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`. 178 179 180PEP 391: Dictionary Based Configuration for Logging 181==================================================== 182 183The :mod:`logging` module provided two kinds of configuration, one style with 184function calls for each option or another style driven by an external file saved 185in a :mod:`ConfigParser` format. Those options did not provide the flexibility 186to create configurations from JSON or YAML files, nor did they support 187incremental configuration, which is needed for specifying logger options from a 188command line. 189 190To support a more flexible style, the module now offers 191:func:`logging.config.dictConfig` for specifying logging configuration with 192plain Python dictionaries. The configuration options include formatters, 193handlers, filters, and loggers. Here's a working example of a configuration 194dictionary:: 195 196 {"version": 1, 197 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, 198 "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"} 199 }, 200 "handlers": {"console": { 201 "class": "logging.StreamHandler", 202 "formatter": "brief", 203 "level": "INFO", 204 "stream": "ext://sys.stdout"}, 205 "console_priority": { 206 "class": "logging.StreamHandler", 207 "formatter": "full", 208 "level": "ERROR", 209 "stream": "ext://sys.stderr"} 210 }, 211 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} 212 213 214If that dictionary is stored in a file called :file:`conf.json`, it can be 215loaded and called with code like this:: 216 217 >>> import json, logging.config 218 >>> with open('conf.json') as f: 219 ... conf = json.load(f) 220 ... 221 >>> logging.config.dictConfig(conf) 222 >>> logging.info("Transaction completed normally") 223 INFO : root : Transaction completed normally 224 >>> logging.critical("Abnormal termination") 225 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination 226 227.. seealso:: 228 229 :pep:`391` - Dictionary Based Configuration for Logging 230 PEP written by Vinay Sajip. 231 232 233PEP 3148: The ``concurrent.futures`` module 234============================================ 235 236Code for creating and managing concurrency is being collected in a new top-level 237namespace, *concurrent*. Its first member is a *futures* package which provides 238a uniform high-level interface for managing threads and processes. 239 240The design for :mod:`concurrent.futures` was inspired by the 241*java.util.concurrent* package. In that model, a running call and its result 242are represented by a :class:`~concurrent.futures.Future` object that abstracts 243features common to threads, processes, and remote procedure calls. That object 244supports status checks (running or done), timeouts, cancellations, adding 245callbacks, and access to results or exceptions. 246 247The primary offering of the new module is a pair of executor classes for 248launching and managing calls. The goal of the executors is to make it easier to 249use existing tools for making parallel calls. They save the effort needed to 250setup a pool of resources, launch the calls, create a results queue, add 251time-out handling, and limit the total number of threads, processes, or remote 252procedure calls. 253 254Ideally, each application should share a single executor across multiple 255components so that process and thread limits can be centrally managed. This 256solves the design challenge that arises when each component has its own 257competing strategy for resource management. 258 259Both classes share a common interface with three methods: 260:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and 261returning a :class:`~concurrent.futures.Future` object; 262:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls 263at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing 264resources. The class is a :term:`context manager` and can be used in a 265:keyword:`with` statement to assure that resources are automatically released 266when currently pending futures are done executing. 267 268A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a 269launch of four parallel threads for copying files:: 270 271 import concurrent.futures, shutil 272 with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: 273 e.submit(shutil.copy, 'src1.txt', 'dest1.txt') 274 e.submit(shutil.copy, 'src2.txt', 'dest2.txt') 275 e.submit(shutil.copy, 'src3.txt', 'dest3.txt') 276 e.submit(shutil.copy, 'src3.txt', 'dest4.txt') 277 278.. seealso:: 279 280 :pep:`3148` - Futures -- Execute Computations Asynchronously 281 PEP written by Brian Quinlan. 282 283 :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an 284 example using threads to fetch multiple web pages in parallel. 285 286 :ref:`Code for computing prime numbers in 287 parallel<processpoolexecutor-example>`, an example demonstrating 288 :class:`~concurrent.futures.ProcessPoolExecutor`. 289 290 291PEP 3147: PYC Repository Directories 292===================================== 293 294Python's scheme for caching bytecode in *.pyc* files did not work well in 295environments with multiple Python interpreters. If one interpreter encountered 296a cached file created by another interpreter, it would recompile the source and 297overwrite the cached file, thus losing the benefits of caching. 298 299The issue of "pyc fights" has become more pronounced as it has become 300commonplace for Linux distributions to ship with multiple versions of Python. 301These conflicts also arise with CPython alternatives such as Unladen Swallow. 302 303To solve this problem, Python's import machinery has been extended to use 304distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and 305Unladen Swallow each competing for a file called "mymodule.pyc", they will now 306look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and 307"mymodule.unladen10.pyc". And to prevent all of these new files from 308cluttering source directories, the *pyc* files are now collected in a 309"__pycache__" directory stored under the package directory. 310 311Aside from the filenames and target directories, the new scheme has a few 312aspects that are visible to the programmer: 313 314* Imported modules now have a :attr:`__cached__` attribute which stores the name 315 of the actual file that was imported: 316 317 >>> import collections 318 >>> collections.__cached__ # doctest: +SKIP 319 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 320 321* The tag that is unique to each interpreter is accessible from the :mod:`imp` 322 module: 323 324 >>> import imp 325 >>> imp.get_tag() # doctest: +SKIP 326 'cpython-32' 327 328* Scripts that try to deduce source filename from the imported file now need to 329 be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" 330 filename. Instead, use the new functions in the :mod:`imp` module: 331 332 >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 333 'c:/py32/lib/collections.py' 334 >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP 335 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 336 337* The :mod:`py_compile` and :mod:`compileall` modules have been updated to 338 reflect the new naming convention and target directory. The command-line 339 invocation of *compileall* has new options: ``-i`` for 340 specifying a list of files and directories to compile and ``-b`` which causes 341 bytecode files to be written to their legacy location rather than 342 *__pycache__*. 343 344* The :mod:`importlib.abc` module has been updated with new :term:`abstract base 345 classes <abstract base class>` for loading bytecode files. The obsolete 346 ABCs, :class:`~importlib.abc.PyLoader` and 347 :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how 348 to stay Python 3.1 compatible are included with the documentation). 349 350.. seealso:: 351 352 :pep:`3147` - PYC Repository Directories 353 PEP written by Barry Warsaw. 354 355 356PEP 3149: ABI Version Tagged .so Files 357====================================== 358 359The PYC repository directory allows multiple bytecode cache files to be 360co-located. This PEP implements a similar mechanism for shared object files by 361giving them a common directory and distinct names for each version. 362 363The common directory is "pyshared" and the file names are made distinct by 364identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the 365major and minor version numbers, and optional build flags (such as "d" for 366debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", 367you may see these files when the distribution package is installed:: 368 369 /usr/share/pyshared/foo.cpython-32m.so 370 /usr/share/pyshared/foo.cpython-33md.so 371 372In Python itself, the tags are accessible from functions in the :mod:`sysconfig` 373module:: 374 375 >>> import sysconfig 376 >>> sysconfig.get_config_var('SOABI') # find the version tag 377 'cpython-32mu' 378 >>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension 379 '.cpython-32mu.so' 380 381.. seealso:: 382 383 :pep:`3149` - ABI Version Tagged .so Files 384 PEP written by Barry Warsaw. 385 386 387PEP 3333: Python Web Server Gateway Interface v1.0.1 388===================================================== 389 390This informational PEP clarifies how bytes/text issues are to be handled by the 391WSGI protocol. The challenge is that string handling in Python 3 is most 392conveniently handled with the :class:`str` type even though the HTTP protocol 393is itself bytes oriented. 394 395The PEP differentiates so-called *native strings* that are used for 396request/response headers and metadata versus *byte strings* which are used for 397the bodies of requests and responses. 398 399The *native strings* are always of type :class:`str` but are restricted to code 400points between *U+0000* through *U+00FF* which are translatable to bytes using 401*Latin-1* encoding. These strings are used for the keys and values in the 402environment dictionary and for response headers and statuses in the 403:func:`start_response` function. They must follow :rfc:`2616` with respect to 404encoding. That is, they must either be *ISO-8859-1* characters or use 405:rfc:`2047` MIME encoding. 406 407For developers porting WSGI applications from Python 2, here are the salient 408points: 409 410* If the app already used strings for headers in Python 2, no change is needed. 411 412* If instead, the app encoded output headers or decoded input headers, then the 413 headers will need to be re-encoded to Latin-1. For example, an output header 414 encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from 415 bytes to native strings using ``h.encode('utf-8').decode('latin-1')``. 416 417* Values yielded by an application or sent using the :meth:`write` method 418 must be byte strings. The :func:`start_response` function and environ 419 must use native strings. The two cannot be mixed. 420 421For server implementers writing CGI-to-WSGI pathways or other CGI-style 422protocols, the users must to be able access the environment using native strings 423even though the underlying platform may have a different convention. To bridge 424this gap, the :mod:`wsgiref` module has a new function, 425:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from 426:attr:`os.environ` into native strings and returning a new dictionary. 427 428.. seealso:: 429 430 :pep:`3333` - Python Web Server Gateway Interface v1.0.1 431 PEP written by Phillip Eby. 432 433 434Other Language Changes 435====================== 436 437Some smaller changes made to the core Python language are: 438 439* String formatting for :func:`format` and :meth:`str.format` gained new 440 capabilities for the format character **#**. Previously, for integers in 441 binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', 442 '0o', or '0x' respectively. Now it can also handle floats, complex, and 443 Decimal, causing the output to always have a decimal point even when no digits 444 follow it. 445 446 >>> format(20, '#o') 447 '0o24' 448 >>> format(12.34, '#5.0f') 449 ' 12.' 450 451 (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.) 452 453* There is also a new :meth:`str.format_map` method that extends the 454 capabilities of the existing :meth:`str.format` method by accepting arbitrary 455 :term:`mapping` objects. This new method makes it possible to use string 456 formatting with any of Python's many dictionary-like objects such as 457 :class:`~collections.defaultdict`, :class:`~shelve.Shelf`, 458 :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with 459 custom :class:`dict` subclasses that normalize keys before look-up or that 460 supply a :meth:`__missing__` method for unknown keys:: 461 462 >>> import shelve 463 >>> d = shelve.open('tmp.shl') 464 >>> 'The {project_name} status is {status} as of {date}'.format_map(d) 465 'The testing project status is green as of February 15, 2011' 466 467 >>> class LowerCasedDict(dict): 468 ... def __getitem__(self, key): 469 ... return dict.__getitem__(self, key.lower()) 470 >>> lcd = LowerCasedDict(part='widgets', quantity=10) 471 >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) 472 'There are 10 widgets in stock' 473 474 >>> class PlaceholderDict(dict): 475 ... def __missing__(self, key): 476 ... return '<{}>'.format(key) 477 >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) 478 'Hello <name>, welcome to <location>' 479 480 (Suggested by Raymond Hettinger and implemented by Eric Smith in 481 :issue:`6081`.) 482 483* The interpreter can now be started with a quiet option, ``-q``, to prevent 484 the copyright and version information from being displayed in the interactive 485 mode. The option can be introspected using the :attr:`sys.flags` attribute: 486 487 .. code-block:: shell-session 488 489 $ python -q 490 >>> sys.flags 491 sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, 492 optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, 493 ignore_environment=0, verbose=0, bytes_warning=0, quiet=1) 494 495 (Contributed by Marcin Wojdyr in :issue:`1772833`). 496 497* The :func:`hasattr` function works by calling :func:`getattr` and detecting 498 whether an exception is raised. This technique allows it to detect methods 499 created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which 500 would otherwise be absent from the class dictionary. Formerly, *hasattr* 501 would catch any exception, possibly masking genuine errors. Now, *hasattr* 502 has been tightened to only catch :exc:`AttributeError` and let other 503 exceptions pass through:: 504 505 >>> class A: 506 ... @property 507 ... def f(self): 508 ... return 1 // 0 509 ... 510 >>> a = A() 511 >>> hasattr(a, 'f') 512 Traceback (most recent call last): 513 ... 514 ZeroDivisionError: integer division or modulo by zero 515 516 (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) 517 518* The :func:`str` of a float or complex number is now the same as its 519 :func:`repr`. Previously, the :func:`str` form was shorter but that just 520 caused confusion and is no longer needed now that the shortest possible 521 :func:`repr` is displayed by default: 522 523 >>> import math 524 >>> repr(math.pi) 525 '3.141592653589793' 526 >>> str(math.pi) 527 '3.141592653589793' 528 529 (Proposed and implemented by Mark Dickinson; :issue:`9337`.) 530 531* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method 532 and they also now support the context management protocol. This allows timely 533 release of any resources that were acquired when requesting a buffer from the 534 original object. 535 536 >>> with memoryview(b'abcdefgh') as v: 537 ... print(v.tolist()) 538 [97, 98, 99, 100, 101, 102, 103, 104] 539 540 (Added by Antoine Pitrou; :issue:`9757`.) 541 542* Previously it was illegal to delete a name from the local namespace if it 543 occurs as a free variable in a nested block:: 544 545 def outer(x): 546 def inner(): 547 return x 548 inner() 549 del x 550 551 This is now allowed. Remember that the target of an :keyword:`except` clause 552 is cleared, so this code which used to work with Python 2.6, raised a 553 :exc:`SyntaxError` with Python 3.1 and now works again:: 554 555 def f(): 556 def print_error(): 557 print(e) 558 try: 559 something 560 except Exception as e: 561 print_error() 562 # implicit "del e" here 563 564 (See :issue:`4617`.) 565 566* The internal :c:type:`structsequence` tool now creates subclasses of tuple. 567 This means that C structures like those returned by :func:`os.stat`, 568 :func:`time.gmtime`, and :attr:`sys.version_info` now work like a 569 :term:`named tuple` and now work with functions and methods that 570 expect a tuple as an argument. This is a big step forward in making the C 571 structures as flexible as their pure Python counterparts: 572 573 >>> import sys 574 >>> isinstance(sys.version_info, tuple) 575 True 576 >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP 577 'Version 3.2.0 final(0)' 578 579 (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented 580 by Benjamin Peterson in :issue:`8413`.) 581 582* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` 583 environment variable as an alternative to using ``-W`` at the command line: 584 585 .. code-block:: shell-session 586 587 $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' 588 589 (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.) 590 591* A new warning category, :exc:`ResourceWarning`, has been added. It is 592 emitted when potential issues with resource consumption or cleanup 593 are detected. It is silenced by default in normal release builds but 594 can be enabled through the means provided by the :mod:`warnings` 595 module, or on the command line. 596 597 A :exc:`ResourceWarning` is issued at interpreter shutdown if the 598 :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is 599 set, all uncollectable objects are printed. This is meant to make the 600 programmer aware that their code contains object finalization issues. 601 602 A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed 603 without having been explicitly closed. While the deallocator for such 604 object ensures it closes the underlying operating system resource 605 (usually, a file descriptor), the delay in deallocating the object could 606 produce various issues, especially under Windows. Here is an example 607 of enabling the warning from the command line: 608 609 .. code-block:: shell-session 610 611 $ python -q -Wdefault 612 >>> f = open("foo", "wb") 613 >>> del f 614 __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> 615 616 (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.) 617 618* :class:`range` objects now support *index* and *count* methods. This is part 619 of an effort to make more objects fully implement the 620 :class:`collections.Sequence` :term:`abstract base class`. As a result, the 621 language will have a more uniform API. In addition, :class:`range` objects 622 now support slicing and negative indices, even with values larger than 623 :attr:`sys.maxsize`. This makes *range* more interoperable with lists:: 624 625 >>> range(0, 100, 2).count(10) 626 1 627 >>> range(0, 100, 2).index(10) 628 5 629 >>> range(0, 100, 2)[5] 630 10 631 >>> range(0, 100, 2)[0:5] 632 range(0, 10, 2) 633 634 (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky 635 in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.) 636 637* The :func:`callable` builtin function from Py2.x was resurrected. It provides 638 a concise, readable alternative to using an :term:`abstract base class` in an 639 expression like ``isinstance(x, collections.Callable)``: 640 641 >>> callable(max) 642 True 643 >>> callable(20) 644 False 645 646 (See :issue:`10518`.) 647 648* Python's import mechanism can now load modules installed in directories with 649 non-ASCII characters in the path name. This solved an aggravating problem 650 with home directories for users with non-ASCII characters in their usernames. 651 652 (Required extensive work by Victor Stinner in :issue:`9425`.) 653 654 655New, Improved, and Deprecated Modules 656===================================== 657 658Python's standard library has undergone significant maintenance efforts and 659quality improvements. 660 661The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox` 662module, and :mod:`nntplib` modules now work correctly with the bytes/text model 663in Python 3. For the first time, there is correct handling of messages with 664mixed encodings. 665 666Throughout the standard library, there has been more careful attention to 667encodings and text versus bytes issues. In particular, interactions with the 668operating system are now better able to exchange non-ASCII data using the 669Windows MBCS encoding, locale-aware encodings, or UTF-8. 670 671Another significant win is the addition of substantially better support for 672*SSL* connections and security certificates. 673 674In addition, more classes now implement a :term:`context manager` to support 675convenient and reliable resource clean-up using a :keyword:`with` statement. 676 677email 678----- 679 680The usability of the :mod:`email` package in Python 3 has been mostly fixed by 681the extensive efforts of R. David Murray. The problem was that emails are 682typically read and stored in the form of :class:`bytes` rather than :class:`str` 683text, and they may contain multiple encodings within a single email. So, the 684email package had to be extended to parse and generate email messages in bytes 685format. 686 687* New functions :func:`~email.message_from_bytes` and 688 :func:`~email.message_from_binary_file`, and new classes 689 :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` 690 allow binary message data to be parsed into model objects. 691 692* Given bytes input to the model, :meth:`~email.message.Message.get_payload` 693 will by default decode a message body that has a 694 :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset 695 specified in the MIME headers and return the resulting string. 696 697* Given bytes input to the model, :class:`~email.generator.Generator` will 698 convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of 699 *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`. 700 701 Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded 702 using the *unknown-8bit* character set. 703 704* A new class :class:`~email.generator.BytesGenerator` produces bytes as output, 705 preserving any unchanged non-ASCII data that was present in the input used to 706 build the model, including message bodies with a 707 :mailheader:`Content-Transfer-Encoding` of *8bit*. 708 709* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string 710 for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, 711 and a new method, :meth:`~smtplib.SMTP.send_message` accepts a 712 :class:`~email.message.Message` object and can optionally obtain the 713 *from_addr* and *to_addrs* addresses directly from the object. 714 715(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.) 716 717elementtree 718----------- 719 720The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree` 721counterpart have been updated to version 1.3. 722 723Several new and useful functions and methods have been added: 724 725* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document 726 from a sequence of fragments 727* :func:`xml.etree.ElementTree.register_namespace` for registering a global 728 namespace prefix 729* :func:`xml.etree.ElementTree.tostringlist` for string representation 730 including all sublists 731* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero 732 or more elements 733* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and 734 subelements 735* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over 736 an element and its subelements 737* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element 738* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype 739 declaration 740 741Two methods have been deprecated: 742 743* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead. 744* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead. 745 746For details of the update, see `Introducing ElementTree 747<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website. 748 749(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.) 750 751functools 752--------- 753 754* The :mod:`functools` module includes a new decorator for caching function 755 calls. :func:`functools.lru_cache` can save repeated queries to an external 756 resource whenever the results are expected to be the same. 757 758 For example, adding a caching decorator to a database query function can save 759 database accesses for popular searches: 760 761 >>> import functools 762 >>> @functools.lru_cache(maxsize=300) 763 ... def get_phone_number(name): 764 ... c = conn.cursor() 765 ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) 766 ... return c.fetchone()[0] 767 768 >>> for name in user_requests: # doctest: +SKIP 769 ... get_phone_number(name) # cached lookup 770 771 To help with choosing an effective cache size, the wrapped function is 772 instrumented for tracking cache statistics: 773 774 >>> get_phone_number.cache_info() # doctest: +SKIP 775 CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) 776 777 If the phonelist table gets updated, the outdated contents of the cache can be 778 cleared with: 779 780 >>> get_phone_number.cache_clear() 781 782 (Contributed by Raymond Hettinger and incorporating design ideas from Jim 783 Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 784 <https://code.activestate.com/recipes/498245>`_\, `recipe 577479 785 <https://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and 786 :issue:`10593`.) 787 788* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute 789 pointing to the original callable function. This allows wrapped functions to 790 be introspected. It also copies :attr:`__annotations__` if defined. And now 791 it also gracefully skips over missing attributes such as :attr:`__doc__` which 792 might not be defined for the wrapped callable. 793 794 In the above example, the cache can be removed by recovering the original 795 function: 796 797 >>> get_phone_number = get_phone_number.__wrapped__ # uncached function 798 799 (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and 800 :issue:`8814`.) 801 802* To help write classes with rich comparison methods, a new decorator 803 :func:`functools.total_ordering` will use existing equality and inequality 804 methods to fill in the remaining methods. 805 806 For example, supplying *__eq__* and *__lt__* will enable 807 :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*:: 808 809 @total_ordering 810 class Student: 811 def __eq__(self, other): 812 return ((self.lastname.lower(), self.firstname.lower()) == 813 (other.lastname.lower(), other.firstname.lower())) 814 815 def __lt__(self, other): 816 return ((self.lastname.lower(), self.firstname.lower()) < 817 (other.lastname.lower(), other.firstname.lower())) 818 819 With the *total_ordering* decorator, the remaining comparison methods 820 are filled in automatically. 821 822 (Contributed by Raymond Hettinger.) 823 824* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key` 825 function converts an old-style comparison function to 826 modern :term:`key function`: 827 828 >>> # locale-aware sort order 829 >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP 830 831 For sorting examples and a brief sorting tutorial, see the `Sorting HowTo 832 <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial. 833 834 (Contributed by Raymond Hettinger.) 835 836itertools 837--------- 838 839* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function 840 modeled on APL's *scan* operator and Numpy's *accumulate* function: 841 842 >>> from itertools import accumulate 843 >>> list(accumulate([8, 2, 50])) 844 [8, 10, 60] 845 846 >>> prob_dist = [0.1, 0.4, 0.2, 0.3] 847 >>> list(accumulate(prob_dist)) # cumulative probability distribution 848 [0.1, 0.5, 0.7, 1.0] 849 850 For an example using :func:`~itertools.accumulate`, see the :ref:`examples for 851 the random module <random-examples>`. 852 853 (Contributed by Raymond Hettinger and incorporating design suggestions 854 from Mark Dickinson.) 855 856collections 857----------- 858 859* The :class:`collections.Counter` class now has two forms of in-place 860 subtraction, the existing *-=* operator for `saturating subtraction 861 <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new 862 :meth:`~collections.Counter.subtract` method for regular subtraction. The 863 former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_ 864 which only have positive counts, and the latter is more suitable for use cases 865 that allow negative counts: 866 867 >>> from collections import Counter 868 >>> tally = Counter(dogs=5, cats=3) 869 >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction 870 >>> tally 871 Counter({'dogs': 3}) 872 873 >>> tally = Counter(dogs=5, cats=3) 874 >>> tally.subtract(dogs=2, cats=8) # regular subtraction 875 >>> tally 876 Counter({'dogs': 3, 'cats': -5}) 877 878 (Contributed by Raymond Hettinger.) 879 880* The :class:`collections.OrderedDict` class has a new method 881 :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and 882 moves it to either the first or last position in the ordered sequence. 883 884 The default is to move an item to the last position. This is equivalent of 885 renewing an entry with ``od[k] = od.pop(k)``. 886 887 A fast move-to-end operation is useful for resequencing entries. For example, 888 an ordered dictionary can be used to track order of access by aging entries 889 from the oldest to the most recently accessed. 890 891 >>> from collections import OrderedDict 892 >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) 893 >>> list(d) 894 ['a', 'b', 'X', 'd', 'e'] 895 >>> d.move_to_end('X') 896 >>> list(d) 897 ['a', 'b', 'd', 'e', 'X'] 898 899 (Contributed by Raymond Hettinger.) 900 901* The :class:`collections.deque` class grew two new methods 902 :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that 903 make them more substitutable for :class:`list` objects: 904 905 >>> from collections import deque 906 >>> d = deque('simsalabim') 907 >>> d.count('s') 908 2 909 >>> d.reverse() 910 >>> d 911 deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's']) 912 913 (Contributed by Raymond Hettinger.) 914 915threading 916--------- 917 918The :mod:`threading` module has a new :class:`~threading.Barrier` 919synchronization class for making multiple threads wait until all of them have 920reached a common barrier point. Barriers are useful for making sure that a task 921with multiple preconditions does not run until all of the predecessor tasks are 922complete. 923 924Barriers can work with an arbitrary number of threads. This is a generalization 925of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which 926is defined for only two threads. 927 928Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects 929are suitable for use in loops. The separate *filling* and *draining* phases 930assure that all threads get released (drained) before any one of them can loop 931back and re-enter the barrier. The barrier fully resets after each cycle. 932 933Example of using barriers:: 934 935 from threading import Barrier, Thread 936 937 def get_votes(site): 938 ballots = conduct_election(site) 939 all_polls_closed.wait() # do not count until all polls are closed 940 totals = summarize(ballots) 941 publish(site, totals) 942 943 all_polls_closed = Barrier(len(sites)) 944 for site in sites: 945 Thread(target=get_votes, args=(site,)).start() 946 947In this example, the barrier enforces a rule that votes cannot be counted at any 948polling site until all polls are closed. Notice how a solution with a barrier 949is similar to one with :meth:`threading.Thread.join`, but the threads stay alive 950and continue to do work (summarizing ballots) after the barrier point is 951crossed. 952 953If any of the predecessor tasks can hang or be delayed, a barrier can be created 954with an optional *timeout* parameter. Then if the timeout period elapses before 955all the predecessor tasks reach the barrier point, all waiting threads are 956released and a :exc:`~threading.BrokenBarrierError` exception is raised:: 957 958 def get_votes(site): 959 ballots = conduct_election(site) 960 try: 961 all_polls_closed.wait(timeout=midnight - time.now()) 962 except BrokenBarrierError: 963 lockbox = seal_ballots(ballots) 964 queue.put(lockbox) 965 else: 966 totals = summarize(ballots) 967 publish(site, totals) 968 969In this example, the barrier enforces a more robust rule. If some election 970sites do not finish before midnight, the barrier times-out and the ballots are 971sealed and deposited in a queue for later handling. 972 973See `Barrier Synchronization Patterns 974<http://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_ 975for more examples of how barriers can be used in parallel computing. Also, there is 976a simple but thorough explanation of barriers in `The Little Book of Semaphores 977<https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*. 978 979(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in 980:issue:`8777`.) 981 982datetime and time 983----------------- 984 985* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that 986 implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC 987 offset and timezone name. This makes it easier to create timezone-aware 988 datetime objects:: 989 990 >>> from datetime import datetime, timezone 991 992 >>> datetime.now(timezone.utc) 993 datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc) 994 995 >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z") 996 datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc) 997 998* Also, :class:`~datetime.timedelta` objects can now be multiplied by 999 :class:`float` and divided by :class:`float` and :class:`int` objects. 1000 And :class:`~datetime.timedelta` objects can now divide one another. 1001 1002* The :meth:`datetime.date.strftime` method is no longer restricted to years 1003 after 1900. The new supported year range is from 1000 to 9999 inclusive. 1004 1005* Whenever a two-digit year is used in a time tuple, the interpretation has been 1006 governed by :attr:`time.accept2dyear`. The default is ``True`` which means that 1007 for a two-digit year, the century is guessed according to the POSIX rules 1008 governing the ``%y`` strptime format. 1009 1010 Starting with Py3.2, use of the century guessing heuristic will emit a 1011 :exc:`DeprecationWarning`. Instead, it is recommended that 1012 :attr:`time.accept2dyear` be set to ``False`` so that large date ranges 1013 can be used without guesswork:: 1014 1015 >>> import time, warnings 1016 >>> warnings.resetwarnings() # remove the default warning filters 1017 1018 >>> time.accept2dyear = True # guess whether 11 means 11 or 2011 1019 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1020 Warning (from warnings module): 1021 ... 1022 DeprecationWarning: Century info guessed for a 2-digit year. 1023 'Fri Jan 1 12:34:56 2011' 1024 1025 >>> time.accept2dyear = False # use the full range of allowable dates 1026 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1027 'Fri Jan 1 12:34:56 11' 1028 1029 Several functions now have significantly expanded date ranges. When 1030 :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will 1031 accept any year that fits in a C int, while the :func:`time.mktime` and 1032 :func:`time.strftime` functions will accept the full range supported by the 1033 corresponding operating system functions. 1034 1035(Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`, 1036:issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`, 1037and :issue:`10827`.) 1038 1039.. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity 1040 1041math 1042---- 1043 1044The :mod:`math` module has been updated with six new functions inspired by the 1045C99 standard. 1046 1047The :func:`~math.isfinite` function provides a reliable and fast way to detect 1048special values. It returns ``True`` for regular numbers and ``False`` for *Nan* or 1049*Infinity*: 1050 1051>>> from math import isfinite 1052>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] 1053[True, True, False, False] 1054 1055The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x* 1056without incurring the loss of precision that usually accompanies the subtraction 1057of nearly equal quantities: 1058 1059>>> from math import expm1 1060>>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x 10610.013765762467652909 1062 1063The :func:`~math.erf` function computes a probability integral or `Gaussian 1064error function <https://en.wikipedia.org/wiki/Error_function>`_. The 1065complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: 1066 1067.. doctest:: 1068 :options: +SKIP 1069 1070 >>> from math import erf, erfc, sqrt 1071 >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation 1072 0.682689492137086 1073 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation 1074 0.31731050786291404 1075 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) 1076 1.0 1077 1078The :func:`~math.gamma` function is a continuous extension of the factorial 1079function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because 1080the function is related to factorials, it grows large even for small values of 1081*x*, so there is also a :func:`~math.lgamma` function for computing the natural 1082logarithm of the gamma function: 1083 1084>>> from math import gamma, lgamma 1085>>> gamma(7.0) # six factorial 1086720.0 1087>>> lgamma(801.0) # log(800 factorial) 10884551.950730698041 1089 1090(Contributed by Mark Dickinson.) 1091 1092abc 1093--- 1094 1095The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and 1096:func:`~abc.abstractstaticmethod`. 1097 1098These tools make it possible to define an :term:`abstract base class` that 1099requires a particular :func:`classmethod` or :func:`staticmethod` to be 1100implemented:: 1101 1102 class Temperature(metaclass=abc.ABCMeta): 1103 @abc.abstractclassmethod 1104 def from_fahrenheit(cls, t): 1105 ... 1106 @abc.abstractclassmethod 1107 def from_celsius(cls, t): 1108 ... 1109 1110(Patch submitted by Daniel Urban; :issue:`5867`.) 1111 1112io 1113-- 1114 1115The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which 1116provides functionality similar to :func:`memoryview`. It creates an editable 1117view of the data without making a copy. The buffer's random access and support 1118for slice notation are well-suited to in-place editing:: 1119 1120 >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 1121 1122 >>> def change_location(buffer, record_number, location): 1123 ... start = record_number * REC_LEN + LOC_START 1124 ... buffer[start: start+LOC_LEN] = location 1125 1126 >>> import io 1127 1128 >>> byte_stream = io.BytesIO( 1129 ... b'G3805 storeroom Main chassis ' 1130 ... b'X7899 shipping Reserve cog ' 1131 ... b'L6988 receiving Primary sprocket' 1132 ... ) 1133 >>> buffer = byte_stream.getbuffer() 1134 >>> change_location(buffer, 1, b'warehouse ') 1135 >>> change_location(buffer, 0, b'showroom ') 1136 >>> print(byte_stream.getvalue()) 1137 b'G3805 showroom Main chassis ' 1138 b'X7899 warehouse Reserve cog ' 1139 b'L6988 receiving Primary sprocket' 1140 1141(Contributed by Antoine Pitrou in :issue:`5506`.) 1142 1143reprlib 1144------- 1145 1146When writing a :meth:`__repr__` method for a custom container, it is easy to 1147forget to handle the case where a member refers back to the container itself. 1148Python's builtin objects such as :class:`list` and :class:`set` handle 1149self-reference by displaying "..." in the recursive part of the representation 1150string. 1151 1152To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new 1153decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to 1154:meth:`__repr__` and substituting a placeholder string instead:: 1155 1156 >>> class MyList(list): 1157 ... @recursive_repr() 1158 ... def __repr__(self): 1159 ... return '<' + '|'.join(map(repr, self)) + '>' 1160 ... 1161 >>> m = MyList('abc') 1162 >>> m.append(m) 1163 >>> m.append('x') 1164 >>> print(m) 1165 <'a'|'b'|'c'|...|'x'> 1166 1167(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.) 1168 1169logging 1170------- 1171 1172In addition to dictionary-based configuration described above, the 1173:mod:`logging` package has many other improvements. 1174 1175The logging documentation has been augmented by a :ref:`basic tutorial 1176<logging-basic-tutorial>`\, an :ref:`advanced tutorial 1177<logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of 1178logging recipes. These documents are the fastest way to learn about logging. 1179 1180The :func:`logging.basicConfig` set-up function gained a *style* argument to 1181support three different types of string formatting. It defaults to "%" for 1182traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or 1183can be set to "$" for the shell-style formatting provided by 1184:class:`string.Template`. The following three configurations are equivalent:: 1185 1186 >>> from logging import basicConfig 1187 >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s") 1188 >>> basicConfig(style='{', format="{name} -> {levelname} {message}") 1189 >>> basicConfig(style='$', format="$name -> $levelname: $message") 1190 1191If no configuration is set-up before a logging event occurs, there is now a 1192default configuration using a :class:`~logging.StreamHandler` directed to 1193:attr:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an 1194event occurring before a configuration was set-up would either raise an 1195exception or silently drop the event depending on the value of 1196:attr:`logging.raiseExceptions`. The new default handler is stored in 1197:attr:`logging.lastResort`. 1198 1199The use of filters has been simplified. Instead of creating a 1200:class:`~logging.Filter` object, the predicate can be any Python callable that 1201returns ``True`` or ``False``. 1202 1203There were a number of other improvements that add flexibility and simplify 1204configuration. See the module documentation for a full listing of changes in 1205Python 3.2. 1206 1207csv 1208--- 1209 1210The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`, 1211which applies quoting for all fields and a traditional Unix style with ``'\n'`` as 1212the line terminator. The registered dialect name is ``unix``. 1213 1214The :class:`csv.DictWriter` has a new method, 1215:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document 1216the field names:: 1217 1218 >>> import csv, sys 1219 >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix') 1220 >>> w.writeheader() 1221 "name","dept" 1222 >>> w.writerows([ 1223 ... {'name': 'tom', 'dept': 'accounting'}, 1224 ... {'name': 'susan', 'dept': 'Salesl'}]) 1225 "tom","accounting" 1226 "susan","sales" 1227 1228(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method 1229suggested by Ed Abraham in :issue:`1537721`.) 1230 1231contextlib 1232---------- 1233 1234There is a new and slightly mind-blowing tool 1235:class:`~contextlib.ContextDecorator` that is helpful for creating a 1236:term:`context manager` that does double duty as a function decorator. 1237 1238As a convenience, this new functionality is used by 1239:func:`~contextlib.contextmanager` so that no extra effort is needed to support 1240both roles. 1241 1242The basic idea is that both context managers and function decorators can be used 1243for pre-action and post-action wrappers. Context managers wrap a group of 1244statements using a :keyword:`with` statement, and function decorators wrap a 1245group of statements enclosed in a function. So, occasionally there is a need to 1246write a pre-action or post-action wrapper that can be used in either role. 1247 1248For example, it is sometimes useful to wrap functions or groups of statements 1249with a logger that can track the time of entry and time of exit. Rather than 1250writing both a function decorator and a context manager for the task, the 1251:func:`~contextlib.contextmanager` provides both capabilities in a single 1252definition:: 1253 1254 from contextlib import contextmanager 1255 import logging 1256 1257 logging.basicConfig(level=logging.INFO) 1258 1259 @contextmanager 1260 def track_entry_and_exit(name): 1261 logging.info('Entering: %s', name) 1262 yield 1263 logging.info('Exiting: %s', name) 1264 1265Formerly, this would have only been usable as a context manager:: 1266 1267 with track_entry_and_exit('widget loader'): 1268 print('Some time consuming activity goes here') 1269 load_widget() 1270 1271Now, it can be used as a decorator as well:: 1272 1273 @track_entry_and_exit('widget loader') 1274 def activity(): 1275 print('Some time consuming activity goes here') 1276 load_widget() 1277 1278Trying to fulfill two roles at once places some limitations on the technique. 1279Context managers normally have the flexibility to return an argument usable by 1280a :keyword:`with` statement, but there is no parallel for function decorators. 1281 1282In the above example, there is not a clean way for the *track_entry_and_exit* 1283context manager to return a logging instance for use in the body of enclosed 1284statements. 1285 1286(Contributed by Michael Foord in :issue:`9110`.) 1287 1288decimal and fractions 1289--------------------- 1290 1291Mark Dickinson crafted an elegant and efficient scheme for assuring that 1292different numeric datatypes will have the same hash value whenever their actual 1293values are equal (:issue:`8188`):: 1294 1295 assert hash(Fraction(3, 2)) == hash(1.5) == \ 1296 hash(Decimal("1.5")) == hash(complex(1.5, 0)) 1297 1298Some of the hashing details are exposed through a new attribute, 1299:attr:`sys.hash_info`, which describes the bit width of the hash value, the 1300prime modulus, the hash values for *infinity* and *nan*, and the multiplier 1301used for the imaginary part of a number: 1302 1303>>> sys.hash_info # doctest: +SKIP 1304sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003) 1305 1306An early decision to limit the inter-operability of various numeric types has 1307been relaxed. It is still unsupported (and ill-advised) to have implicit 1308mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')`` 1309because the latter loses information in the process of constructing the binary 1310float. However, since existing floating point value can be converted losslessly 1311to either a decimal or rational representation, it makes sense to add them to 1312the constructor and to support mixed-type comparisons. 1313 1314* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects 1315 directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float` 1316 method (:issue:`8257`). 1317 1318* Mixed type comparisons are now fully supported so that 1319 :class:`~decimal.Decimal` objects can be directly compared with :class:`float` 1320 and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`). 1321 1322Similar changes were made to :class:`fractions.Fraction` so that the 1323:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal` 1324methods are no longer needed (:issue:`8294`): 1325 1326>>> from decimal import Decimal 1327>>> from fractions import Fraction 1328>>> Decimal(1.1) 1329Decimal('1.100000000000000088817841970012523233890533447265625') 1330>>> Fraction(1.1) 1331Fraction(2476979795053773, 2251799813685248) 1332 1333Another useful change for the :mod:`decimal` module is that the 1334:attr:`Context.clamp` attribute is now public. This is useful in creating 1335contexts that correspond to the decimal interchange formats specified in IEEE 1336754 (see :issue:`8540`). 1337 1338(Contributed by Mark Dickinson and Raymond Hettinger.) 1339 1340ftp 1341--- 1342 1343The :class:`ftplib.FTP` class now supports the context management protocol to 1344unconditionally consume :exc:`socket.error` exceptions and to close the FTP 1345connection when done:: 1346 1347 >>> from ftplib import FTP 1348 >>> with FTP("ftp1.at.proftpd.org") as ftp: 1349 ftp.login() 1350 ftp.dir() 1351 1352 '230 Anonymous login ok, restrictions apply.' 1353 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . 1354 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. 1355 dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS 1356 dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora 1357 1358Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` 1359also grew auto-closing context managers:: 1360 1361 with fileinput.input(files=('log1.txt', 'log2.txt')) as f: 1362 for line in f: 1363 process(line) 1364 1365(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and 1366by Georg Brandl in :issue:`8046` and :issue:`1286`.) 1367 1368The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a 1369:class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1370certificates and private keys into a single (potentially long-lived) structure. 1371 1372(Contributed by Giampaolo Rodolà; :issue:`8806`.) 1373 1374popen 1375----- 1376 1377The :func:`os.popen` and :func:`subprocess.Popen` functions now support 1378:keyword:`with` statements for auto-closing of the file descriptors. 1379 1380(Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and 1381:issue:`10554`.) 1382 1383select 1384------ 1385 1386The :mod:`select` module now exposes a new, constant attribute, 1387:attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are 1388guaranteed not to block when :func:`select.select` says a pipe is ready 1389for writing. 1390 1391>>> import select 1392>>> select.PIPE_BUF # doctest: +SKIP 1393512 1394 1395(Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`) 1396 1397gzip and zipfile 1398---------------- 1399 1400:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` 1401:term:`abstract base class` (except for ``truncate()``). It also has a 1402:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as 1403zero-padded file objects. 1404 1405The :mod:`gzip` module also gains the :func:`~gzip.compress` and 1406:func:`~gzip.decompress` functions for easier in-memory compression and 1407decompression. Keep in mind that text needs to be encoded as :class:`bytes` 1408before compressing and decompressing: 1409 1410>>> import gzip 1411>>> s = 'Three shall be the number thou shalt count, ' 1412>>> s += 'and the number of the counting shall be three' 1413>>> b = s.encode() # convert to utf-8 1414>>> len(b) 141589 1416>>> c = gzip.compress(b) 1417>>> len(c) 141877 1419>>> gzip.decompress(c).decode()[:42] # decompress and convert to text 1420'Three shall be the number thou shalt count' 1421 1422(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir 1423Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and 1424:issue:`2846`.) 1425 1426Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent 1427files stored inside an archive. The new implementation is significantly faster 1428and can be wrapped in an :class:`io.BufferedReader` object for more speedups. It 1429also solves an issue where interleaved calls to *read* and *readline* gave the 1430wrong results. 1431 1432(Patch submitted by Nir Aides in :issue:`7610`.) 1433 1434tarfile 1435------- 1436 1437The :class:`~tarfile.TarFile` class can now be used as a context manager. In 1438addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*, 1439that controls which files are added to the archive and allows the file metadata 1440to be edited. 1441 1442The new *filter* option replaces the older, less flexible *exclude* parameter 1443which is now deprecated. If specified, the optional *filter* parameter needs to 1444be a :term:`keyword argument`. The user-supplied filter function accepts a 1445:class:`~tarfile.TarInfo` object and returns an updated 1446:class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the 1447function can return ``None``:: 1448 1449 >>> import tarfile, glob 1450 1451 >>> def myfilter(tarinfo): 1452 ... if tarinfo.isfile(): # only save real files 1453 ... tarinfo.uname = 'monty' # redact the user name 1454 ... return tarinfo 1455 1456 >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: 1457 ... for filename in glob.glob('*.txt'): 1458 ... tf.add(filename, filter=myfilter) 1459 ... tf.list() 1460 -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt 1461 -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt 1462 -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt 1463 -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt 1464 -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt 1465 1466(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.) 1467 1468hashlib 1469------- 1470 1471The :mod:`hashlib` module has two new constant attributes listing the hashing 1472algorithms guaranteed to be present in all implementations and those available 1473on the current implementation:: 1474 1475 >>> import hashlib 1476 1477 >>> hashlib.algorithms_guaranteed 1478 {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'} 1479 1480 >>> hashlib.algorithms_available 1481 {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256', 1482 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2', 1483 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224', 1484 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'} 1485 1486(Suggested by Carl Chenet in :issue:`7418`.) 1487 1488ast 1489--- 1490 1491The :mod:`ast` module has a wonderful a general-purpose tool for safely 1492evaluating expression strings using the Python literal 1493syntax. The :func:`ast.literal_eval` function serves as a secure alternative to 1494the builtin :func:`eval` function which is easily abused. Python 3.2 adds 1495:class:`bytes` and :class:`set` literals to the list of supported types: 1496strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``. 1497 1498:: 1499 1500 >>> from ast import literal_eval 1501 1502 >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}" 1503 >>> literal_eval(request) 1504 {'args': (2, 0.5), 'req': 3, 'func': 'pow'} 1505 1506 >>> request = "os.system('do something harmful')" 1507 >>> literal_eval(request) 1508 Traceback (most recent call last): 1509 ... 1510 ValueError: malformed node or string: <_ast.Call object at 0x101739a10> 1511 1512(Implemented by Benjamin Peterson and Georg Brandl.) 1513 1514os 1515-- 1516 1517Different operating systems use various encodings for filenames and environment 1518variables. The :mod:`os` module provides two new functions, 1519:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding 1520filenames: 1521 1522>>> import os 1523>>> filename = 'Sehenswürdigkeiten' 1524>>> os.fsencode(filename) 1525b'Sehensw\xc3\xbcrdigkeiten' 1526 1527Some operating systems allow direct access to encoded bytes in the 1528environment. If so, the :attr:`os.supports_bytes_environ` constant will be 1529true. 1530 1531For direct access to encoded environment variables (if available), 1532use the new :func:`os.getenvb` function or use :data:`os.environb` 1533which is a bytes version of :data:`os.environ`. 1534 1535(Contributed by Victor Stinner.) 1536 1537shutil 1538------ 1539 1540The :func:`shutil.copytree` function has two new options: 1541 1542* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function 1543 copies a file pointed to by a symlink, not the symlink itself. This option 1544 will silence the error raised if the file doesn't exist. 1545 1546* *copy_function*: is a callable that will be used to copy files. 1547 :func:`shutil.copy2` is used by default. 1548 1549(Contributed by Tarek Ziadé.) 1550 1551In addition, the :mod:`shutil` module now supports :ref:`archiving operations 1552<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles, 1553and bzipped tarfiles. And there are functions for registering additional 1554archiving file formats (such as xz compressed tarfiles or custom formats). 1555 1556The principal functions are :func:`~shutil.make_archive` and 1557:func:`~shutil.unpack_archive`. By default, both operate on the current 1558directory (which can be set by :func:`os.chdir`) and on any sub-directories. 1559The archive filename needs to be specified with a full pathname. The archiving 1560step is non-destructive (the original files are left unchanged). 1561 1562:: 1563 1564 >>> import shutil, pprint 1565 1566 >>> os.chdir('mydata') # change to the source directory 1567 >>> f = shutil.make_archive('/var/backup/mydata', 1568 ... 'zip') # archive the current directory 1569 >>> f # show the name of archive 1570 '/var/backup/mydata.zip' 1571 >>> os.chdir('tmp') # change to an unpacking 1572 >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data 1573 1574 >>> pprint.pprint(shutil.get_archive_formats()) # display known formats 1575 [('bztar', "bzip2'ed tar-file"), 1576 ('gztar', "gzip'ed tar-file"), 1577 ('tar', 'uncompressed tar file'), 1578 ('zip', 'ZIP file')] 1579 1580 >>> shutil.register_archive_format( # register a new archive format 1581 ... name='xz', 1582 ... function=xz.compress, # callable archiving function 1583 ... extra_args=[('level', 8)], # arguments to the function 1584 ... description='xz compression' 1585 ... ) 1586 1587(Contributed by Tarek Ziadé.) 1588 1589sqlite3 1590------- 1591 1592The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities. 1593 1594* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an 1595 active transaction for uncommitted changes. 1596 1597* The :meth:`sqlite3.Connection.enable_load_extension` and 1598 :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite 1599 extensions from ".so" files. One well-known extension is the fulltext-search 1600 extension distributed with SQLite. 1601 1602(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) 1603 1604html 1605---- 1606 1607A new :mod:`html` module was introduced with only a single function, 1608:func:`~html.escape`, which is used for escaping reserved characters from HTML 1609markup: 1610 1611>>> import html 1612>>> html.escape('x > 2 && x < 7') 1613'x > 2 && x < 7' 1614 1615socket 1616------ 1617 1618The :mod:`socket` module has two new improvements. 1619 1620* Socket objects now have a :meth:`~socket.socket.detach()` method which puts 1621 the socket into closed state without actually closing the underlying file 1622 descriptor. The latter can then be reused for other purposes. 1623 (Added by Antoine Pitrou; :issue:`8524`.) 1624 1625* :func:`socket.create_connection` now supports the context management protocol 1626 to unconditionally consume :exc:`socket.error` exceptions and to close the 1627 socket when done. 1628 (Contributed by Giampaolo Rodolà; :issue:`9794`.) 1629 1630ssl 1631--- 1632 1633The :mod:`ssl` module added a number of features to satisfy common requirements 1634for secure (encrypted, authenticated) internet connections: 1635 1636* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent 1637 SSL data, such as protocol settings, certificates, private keys, and various 1638 other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating 1639 an SSL socket from an SSL context. 1640 1641* A new function, :func:`ssl.match_hostname`, supports server identity 1642 verification for higher-level protocols by implementing the rules of HTTPS 1643 (from :rfc:`2818`) which are also suitable for other protocols. 1644 1645* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* 1646 argument. The *ciphers* string lists the allowed encryption algorithms using 1647 the format described in the `OpenSSL documentation 1648 <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__. 1649 1650* When linked against recent versions of OpenSSL, the :mod:`ssl` module now 1651 supports the Server Name Indication extension to the TLS protocol, allowing 1652 multiple "virtual hosts" using different certificates on a single IP port. 1653 This extension is only supported in client mode, and is activated by passing 1654 the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`. 1655 1656* Various options have been added to the :mod:`ssl` module, such as 1657 :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2 1658 protocol. 1659 1660* The extension now loads all the OpenSSL ciphers and digest algorithms. If 1661 some SSL certificates cannot be verified, they are reported as an "unknown 1662 algorithm" error. 1663 1664* The version of OpenSSL being used is now accessible using the module 1665 attributes :data:`ssl.OPENSSL_VERSION` (a string), 1666 :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and 1667 :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). 1668 1669(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`, 1670:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.) 1671 1672nntp 1673---- 1674 1675The :mod:`nntplib` module has a revamped implementation with better bytes and 1676text semantics as well as more practical APIs. These improvements break 1677compatibility with the nntplib version in Python 3.1, which was partly 1678dysfunctional in itself. 1679 1680Support for secure connections through both implicit (using 1681:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`) 1682TLS has also been added. 1683 1684(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.) 1685 1686certificates 1687------------ 1688 1689:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` 1690and :func:`urllib.request.urlopen` now take optional arguments to allow for 1691server certificate checking against a set of Certificate Authorities, 1692as recommended in public uses of HTTPS. 1693 1694(Added by Antoine Pitrou, :issue:`9003`.) 1695 1696imaplib 1697------- 1698 1699Support for explicit TLS on standard IMAP4 connections has been added through 1700the new :mod:`imaplib.IMAP4.starttls` method. 1701 1702(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.) 1703 1704http.client 1705----------- 1706 1707There were a number of small API improvements in the :mod:`http.client` module. 1708The old-style HTTP 0.9 simple responses are no longer supported and the *strict* 1709parameter is deprecated in all classes. 1710 1711The :class:`~http.client.HTTPConnection` and 1712:class:`~http.client.HTTPSConnection` classes now have a *source_address* 1713parameter for a (host, port) tuple indicating where the HTTP connection is made 1714from. 1715 1716Support for certificate checking and HTTPS virtual hosts were added to 1717:class:`~http.client.HTTPSConnection`. 1718 1719The :meth:`~http.client.HTTPConnection.request` method on connection objects 1720allowed an optional *body* argument so that a :term:`file object` could be used 1721to supply the content of the request. Conveniently, the *body* argument now 1722also accepts an :term:`iterable` object so long as it includes an explicit 1723``Content-Length`` header. This extended interface is much more flexible than 1724before. 1725 1726To establish an HTTPS connection through a proxy server, there is a new 1727:meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and 1728port for HTTP Connect tunneling. 1729 1730To match the behavior of :mod:`http.server`, the HTTP client library now also 1731encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that 1732for incoming headers, so now the behavior is consistent for both incoming and 1733outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.) 1734 1735unittest 1736-------- 1737 1738The unittest module has a number of improvements supporting test discovery for 1739packages, easier experimentation at the interactive prompt, new testcase 1740methods, improved diagnostic messages for test failures, and better method 1741names. 1742 1743* The command-line call ``python -m unittest`` can now accept file paths 1744 instead of module names for running specific tests (:issue:`10620`). The new 1745 test discovery can find tests within packages, locating any test importable 1746 from the top-level directory. The top-level directory can be specified with 1747 the `-t` option, a pattern for matching files with ``-p``, and a directory to 1748 start discovery with ``-s``: 1749 1750 .. code-block:: shell-session 1751 1752 $ python -m unittest discover -s my_proj_dir -p _test.py 1753 1754 (Contributed by Michael Foord.) 1755 1756* Experimentation at the interactive prompt is now easier because the 1757 :class:`unittest.case.TestCase` class can now be instantiated without 1758 arguments: 1759 1760 >>> from unittest import TestCase 1761 >>> TestCase().assertEqual(pow(2, 3), 8) 1762 1763 (Contributed by Michael Foord.) 1764 1765* The :mod:`unittest` module has two new methods, 1766 :meth:`~unittest.TestCase.assertWarns` and 1767 :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type 1768 is triggered by the code under test:: 1769 1770 with self.assertWarns(DeprecationWarning): 1771 legacy_function('XYZ') 1772 1773 (Contributed by Antoine Pitrou, :issue:`9754`.) 1774 1775 Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to 1776 compare two iterables to determine if their element counts are equal (whether 1777 the same elements are present with the same number of occurrences regardless 1778 of order):: 1779 1780 def test_anagram(self): 1781 self.assertCountEqual('algorithm', 'logarithm') 1782 1783 (Contributed by Raymond Hettinger.) 1784 1785* A principal feature of the unittest module is an effort to produce meaningful 1786 diagnostics when a test fails. When possible, the failure is recorded along 1787 with a diff of the output. This is especially helpful for analyzing log files 1788 of failed test runs. However, since diffs can sometime be voluminous, there is 1789 a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of 1790 diffs displayed. 1791 1792* In addition, the method names in the module have undergone a number of clean-ups. 1793 1794 For example, :meth:`~unittest.TestCase.assertRegex` is the new name for 1795 :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the 1796 test uses :func:`re.search`, not :func:`re.match`. Other methods using 1797 regular expressions are now named using short form "Regex" in preference to 1798 "Regexp" -- this matches the names used in other unittest implementations, 1799 matches Python's old name for the :mod:`re` module, and it has unambiguous 1800 camel-casing. 1801 1802 (Contributed by Raymond Hettinger and implemented by Ezio Melotti.) 1803 1804* To improve consistency, some long-standing method aliases are being 1805 deprecated in favor of the preferred names: 1806 1807 =============================== ============================== 1808 Old Name Preferred Name 1809 =============================== ============================== 1810 :meth:`assert_` :meth:`.assertTrue` 1811 :meth:`assertEquals` :meth:`.assertEqual` 1812 :meth:`assertNotEquals` :meth:`.assertNotEqual` 1813 :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual` 1814 :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual` 1815 =============================== ============================== 1816 1817 Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected 1818 to be removed in Python 3.3. 1819 1820 (Contributed by Ezio Melotti; :issue:`9424`.) 1821 1822* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated 1823 because it was misimplemented with the arguments in the wrong order. This 1824 created hard-to-debug optical illusions where tests like 1825 ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail. 1826 1827 (Contributed by Raymond Hettinger.) 1828 1829random 1830------ 1831 1832The integer methods in the :mod:`random` module now do a better job of producing 1833uniform distributions. Previously, they computed selections with 1834``int(n*random())`` which had a slight bias whenever *n* was not a power of two. 1835Now, multiple selections are made from a range up to the next power of two and a 1836selection is kept only when it falls within the range ``0 <= x < n``. The 1837functions and methods affected are :func:`~random.randrange`, 1838:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and 1839:func:`~random.sample`. 1840 1841(Contributed by Raymond Hettinger; :issue:`9025`.) 1842 1843poplib 1844------ 1845 1846:class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a 1847:class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1848certificates and private keys into a single (potentially long-lived) 1849structure. 1850 1851(Contributed by Giampaolo Rodolà; :issue:`8807`.) 1852 1853asyncore 1854-------- 1855 1856:class:`asyncore.dispatcher` now provides a 1857:meth:`~asyncore.dispatcher.handle_accepted()` method 1858returning a `(sock, addr)` pair which is called when a connection has actually 1859been established with a new remote endpoint. This is supposed to be used as a 1860replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids 1861the user to call :meth:`~asyncore.dispatcher.accept()` directly. 1862 1863(Contributed by Giampaolo Rodolà; :issue:`6706`.) 1864 1865tempfile 1866-------- 1867 1868The :mod:`tempfile` module has a new context manager, 1869:class:`~tempfile.TemporaryDirectory` which provides easy deterministic 1870cleanup of temporary directories:: 1871 1872 with tempfile.TemporaryDirectory() as tmpdirname: 1873 print('created temporary dir:', tmpdirname) 1874 1875(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) 1876 1877inspect 1878------- 1879 1880* The :mod:`inspect` module has a new function 1881 :func:`~inspect.getgeneratorstate` to easily identify the current state of a 1882 generator-iterator:: 1883 1884 >>> from inspect import getgeneratorstate 1885 >>> def gen(): 1886 ... yield 'demo' 1887 >>> g = gen() 1888 >>> getgeneratorstate(g) 1889 'GEN_CREATED' 1890 >>> next(g) 1891 'demo' 1892 >>> getgeneratorstate(g) 1893 'GEN_SUSPENDED' 1894 >>> next(g, None) 1895 >>> getgeneratorstate(g) 1896 'GEN_CLOSED' 1897 1898 (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) 1899 1900* To support lookups without the possibility of activating a dynamic attribute, 1901 the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`. 1902 Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to 1903 change state while it is searching:: 1904 1905 >>> class A: 1906 ... @property 1907 ... def f(self): 1908 ... print('Running') 1909 ... return 10 1910 ... 1911 >>> a = A() 1912 >>> getattr(a, 'f') 1913 Running 1914 10 1915 >>> inspect.getattr_static(a, 'f') 1916 <property object at 0x1022bd788> 1917 1918 (Contributed by Michael Foord.) 1919 1920pydoc 1921----- 1922 1923The :mod:`pydoc` module now provides a much-improved web server interface, as 1924well as a new command-line option ``-b`` to automatically open a browser window 1925to display that server: 1926 1927.. code-block:: shell-session 1928 1929 $ pydoc3.2 -b 1930 1931(Contributed by Ron Adam; :issue:`2001`.) 1932 1933dis 1934--- 1935 1936The :mod:`dis` module gained two new functions for inspecting code, 1937:func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code 1938object information for the supplied function, method, source code string or code 1939object. The former returns a string and the latter prints it:: 1940 1941 >>> import dis, random 1942 >>> dis.show_code(random.choice) 1943 Name: choice 1944 Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py 1945 Argument count: 2 1946 Kw-only arguments: 0 1947 Number of locals: 3 1948 Stack size: 11 1949 Flags: OPTIMIZED, NEWLOCALS, NOFREE 1950 Constants: 1951 0: 'Choose a random element from a non-empty sequence.' 1952 1: 'Cannot choose from an empty sequence' 1953 Names: 1954 0: _randbelow 1955 1: len 1956 2: ValueError 1957 3: IndexError 1958 Variable names: 1959 0: self 1960 1: seq 1961 2: i 1962 1963In addition, the :func:`~dis.dis` function now accepts string arguments 1964so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened 1965to ``dis(s)``:: 1966 1967 >>> dis('3*x+1 if x%2==1 else x//2') 1968 1 0 LOAD_NAME 0 (x) 1969 3 LOAD_CONST 0 (2) 1970 6 BINARY_MODULO 1971 7 LOAD_CONST 1 (1) 1972 10 COMPARE_OP 2 (==) 1973 13 POP_JUMP_IF_FALSE 28 1974 16 LOAD_CONST 2 (3) 1975 19 LOAD_NAME 0 (x) 1976 22 BINARY_MULTIPLY 1977 23 LOAD_CONST 1 (1) 1978 26 BINARY_ADD 1979 27 RETURN_VALUE 1980 >> 28 LOAD_NAME 0 (x) 1981 31 LOAD_CONST 0 (2) 1982 34 BINARY_FLOOR_DIVIDE 1983 35 RETURN_VALUE 1984 1985Taken together, these improvements make it easier to explore how CPython is 1986implemented and to see for yourself what the language syntax does 1987under-the-hood. 1988 1989(Contributed by Nick Coghlan in :issue:`9147`.) 1990 1991dbm 1992--- 1993 1994All database modules now support the :meth:`get` and :meth:`setdefault` methods. 1995 1996(Suggested by Ray Allen in :issue:`9523`.) 1997 1998ctypes 1999------ 2000 2001A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype. 2002 2003site 2004---- 2005 2006The :mod:`site` module has three new functions useful for reporting on the 2007details of a given Python installation. 2008 2009* :func:`~site.getsitepackages` lists all global site-packages directories. 2010 2011* :func:`~site.getuserbase` reports on the user's base directory where data can 2012 be stored. 2013 2014* :func:`~site.getusersitepackages` reveals the user-specific site-packages 2015 directory path. 2016 2017:: 2018 2019 >>> import site 2020 >>> site.getsitepackages() 2021 ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages', 2022 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python', 2023 '/Library/Python/3.2/site-packages'] 2024 >>> site.getuserbase() 2025 '/Users/raymondhettinger/Library/Python/3.2' 2026 >>> site.getusersitepackages() 2027 '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' 2028 2029Conveniently, some of site's functionality is accessible directly from the 2030command-line: 2031 2032.. code-block:: shell-session 2033 2034 $ python -m site --user-base 2035 /Users/raymondhettinger/.local 2036 $ python -m site --user-site 2037 /Users/raymondhettinger/.local/lib/python3.2/site-packages 2038 2039(Contributed by Tarek Ziadé in :issue:`6693`.) 2040 2041sysconfig 2042--------- 2043 2044The new :mod:`sysconfig` module makes it straightforward to discover 2045installation paths and configuration variables that vary across platforms and 2046installations. 2047 2048The module offers access simple access functions for platform and version 2049information: 2050 2051* :func:`~sysconfig.get_platform` returning values like *linux-i586* or 2052 *macosx-10.6-ppc*. 2053* :func:`~sysconfig.get_python_version` returns a Python version string 2054 such as "3.2". 2055 2056It also provides access to the paths and variables corresponding to one of 2057seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, 2058*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*: 2059 2060* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths 2061 for the current installation scheme. 2062* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific 2063 variables. 2064 2065There is also a convenient command-line interface: 2066 2067.. code-block:: doscon 2068 2069 C:\Python32>python -m sysconfig 2070 Platform: "win32" 2071 Python version: "3.2" 2072 Current installation scheme: "nt" 2073 2074 Paths: 2075 data = "C:\Python32" 2076 include = "C:\Python32\Include" 2077 platinclude = "C:\Python32\Include" 2078 platlib = "C:\Python32\Lib\site-packages" 2079 platstdlib = "C:\Python32\Lib" 2080 purelib = "C:\Python32\Lib\site-packages" 2081 scripts = "C:\Python32\Scripts" 2082 stdlib = "C:\Python32\Lib" 2083 2084 Variables: 2085 BINDIR = "C:\Python32" 2086 BINLIBDEST = "C:\Python32\Lib" 2087 EXE = ".exe" 2088 INCLUDEPY = "C:\Python32\Include" 2089 LIBDEST = "C:\Python32\Lib" 2090 SO = ".pyd" 2091 VERSION = "32" 2092 abiflags = "" 2093 base = "C:\Python32" 2094 exec_prefix = "C:\Python32" 2095 platbase = "C:\Python32" 2096 prefix = "C:\Python32" 2097 projectbase = "C:\Python32" 2098 py_version = "3.2" 2099 py_version_nodot = "32" 2100 py_version_short = "3.2" 2101 srcdir = "C:\Python32" 2102 userbase = "C:\Documents and Settings\Raymond\Application Data\Python" 2103 2104(Moved out of Distutils by Tarek Ziadé.) 2105 2106pdb 2107--- 2108 2109The :mod:`pdb` debugger module gained a number of usability improvements: 2110 2111* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a 2112 :file:`.pdbrc` script file. 2113* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands 2114 that continue debugging. 2115* The :class:`Pdb` class constructor now accepts a *nosigint* argument. 2116* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for 2117 listing source code. 2118* New commands: ``display`` and ``undisplay`` for showing or hiding 2119 the value of an expression if it has changed. 2120* New command: ``interact`` for starting an interactive interpreter containing 2121 the global and local names found in the current scope. 2122* Breakpoints can be cleared by breakpoint number. 2123 2124(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.) 2125 2126configparser 2127------------ 2128 2129The :mod:`configparser` module was modified to improve usability and 2130predictability of the default parser and its supported INI syntax. The old 2131:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser` 2132which has in turn been renamed to :class:`~configparser.ConfigParser`. Support 2133for inline comments is now turned off by default and section or option 2134duplicates are not allowed in a single configuration source. 2135 2136Config parsers gained a new API based on the mapping protocol:: 2137 2138 >>> parser = ConfigParser() 2139 >>> parser.read_string(""" 2140 ... [DEFAULT] 2141 ... location = upper left 2142 ... visible = yes 2143 ... editable = no 2144 ... color = blue 2145 ... 2146 ... [main] 2147 ... title = Main Menu 2148 ... color = green 2149 ... 2150 ... [options] 2151 ... title = Options 2152 ... """) 2153 >>> parser['main']['color'] 2154 'green' 2155 >>> parser['main']['editable'] 2156 'no' 2157 >>> section = parser['options'] 2158 >>> section['title'] 2159 'Options' 2160 >>> section['title'] = 'Options (editable: %(editable)s)' 2161 >>> section['title'] 2162 'Options (editable: no)' 2163 2164The new API is implemented on top of the classical API, so custom parser 2165subclasses should be able to use it without modifications. 2166 2167The INI file structure accepted by config parsers can now be customized. Users 2168can specify alternative option/value delimiters and comment prefixes, change the 2169name of the *DEFAULT* section or switch the interpolation syntax. 2170 2171There is support for pluggable interpolation including an additional interpolation 2172handler :class:`~configparser.ExtendedInterpolation`:: 2173 2174 >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) 2175 >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, 2176 ... 'custom': {'prefix': '/usr/local'}}) 2177 >>> parser.read_string(""" 2178 ... [buildout] 2179 ... parts = 2180 ... zope9 2181 ... instance 2182 ... find-links = 2183 ... ${buildout:directory}/downloads/dist 2184 ... 2185 ... [zope9] 2186 ... recipe = plone.recipe.zope9install 2187 ... location = /opt/zope 2188 ... 2189 ... [instance] 2190 ... recipe = plone.recipe.zope9instance 2191 ... zope9-location = ${zope9:location} 2192 ... zope-conf = ${custom:prefix}/etc/zope.conf 2193 ... """) 2194 >>> parser['buildout']['find-links'] 2195 '\n/home/ambv/zope9/downloads/dist' 2196 >>> parser['instance']['zope-conf'] 2197 '/usr/local/etc/zope.conf' 2198 >>> instance = parser['instance'] 2199 >>> instance['zope-conf'] 2200 '/usr/local/etc/zope.conf' 2201 >>> instance['zope9-location'] 2202 '/opt/zope' 2203 2204A number of smaller features were also introduced, like support for specifying 2205encoding in read operations, specifying fallback values for get-functions, or 2206reading directly from dictionaries and strings. 2207 2208(All changes contributed by Łukasz Langa.) 2209 2210.. XXX consider showing a difflib example 2211 2212urllib.parse 2213------------ 2214 2215A number of usability improvements were made for the :mod:`urllib.parse` module. 2216 2217The :func:`~urllib.parse.urlparse` function now supports `IPv6 2218<https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: 2219 2220 >>> import urllib.parse 2221 >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE 2222 ParseResult(scheme='http', 2223 netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', 2224 path='/foo/', 2225 params='', 2226 query='', 2227 fragment='') 2228 2229The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`:: 2230 2231 >>> r = urllib.parse.urldefrag('http://python.org/about/#target') 2232 >>> r 2233 DefragResult(url='http://python.org/about/', fragment='target') 2234 >>> r[0] 2235 'http://python.org/about/' 2236 >>> r.fragment 2237 'target' 2238 2239And, the :func:`~urllib.parse.urlencode` function is now much more flexible, 2240accepting either a string or bytes type for the *query* argument. If it is a 2241string, then the *safe*, *encoding*, and *error* parameters are sent to 2242:func:`~urllib.parse.quote_plus` for encoding:: 2243 2244 >>> urllib.parse.urlencode([ 2245 ... ('type', 'telenovela'), 2246 ... ('name', '¿Dónde Está Elisa?')], 2247 ... encoding='latin-1') 2248 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' 2249 2250As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` 2251functions now accept ASCII-encoded byte strings as input, so long as they are 2252not mixed with regular strings. If ASCII-encoded byte strings are given as 2253parameters, the return types will also be an ASCII-encoded byte strings: 2254 2255 >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE 2256 ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', 2257 path=b'/about/', params=b'', query=b'', fragment=b'') 2258 2259(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`, 2260:issue:`5468`, and :issue:`9873`.) 2261 2262mailbox 2263------- 2264 2265Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has 2266been fixed for Python 3.2. The challenge was that mailbox had been originally 2267designed with a text interface, but email messages are best represented with 2268:class:`bytes` because various parts of a message may have different encodings. 2269 2270The solution harnessed the :mod:`email` package's binary support for parsing 2271arbitrary email messages. In addition, the solution required a number of API 2272changes. 2273 2274As expected, the :meth:`~mailbox.Mailbox.add` method for 2275:class:`mailbox.Mailbox` objects now accepts binary input. 2276 2277:class:`~io.StringIO` and text file input are deprecated. Also, string input 2278will fail early if non-ASCII characters are used. Previously it would fail when 2279the email was processed in a later step. 2280 2281There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file` 2282method now returns a file in the binary mode (where it used to incorrectly set 2283the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes` 2284method that returns a :class:`bytes` representation of a message corresponding 2285to a given *key*. 2286 2287It is still possible to get non-binary output using the old API's 2288:meth:`~mailbox.Mailbox.get_string` method, but that approach 2289is not very useful. Instead, it is best to extract messages from 2290a :class:`~mailbox.Message` object or to load them from binary input. 2291 2292(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an 2293initial patch by Victor Stinner in :issue:`9124`.) 2294 2295turtledemo 2296---------- 2297 2298The demonstration code for the :mod:`turtle` module was moved from the *Demo* 2299directory to main library. It includes over a dozen sample scripts with 2300lively displays. Being on :attr:`sys.path`, it can now be run directly 2301from the command-line: 2302 2303.. code-block:: shell-session 2304 2305 $ python -m turtledemo 2306 2307(Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.) 2308 2309Multi-threading 2310=============== 2311 2312* The mechanism for serializing execution of concurrently running Python threads 2313 (generally known as the :term:`GIL` or Global Interpreter Lock) has 2314 been rewritten. Among the objectives were more predictable switching 2315 intervals and reduced overhead due to lock contention and the number of 2316 ensuing system calls. The notion of a "check interval" to allow thread 2317 switches has been abandoned and replaced by an absolute duration expressed in 2318 seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. 2319 It currently defaults to 5 milliseconds. 2320 2321 Additional details about the implementation can be read from a `python-dev 2322 mailing-list message 2323 <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_ 2324 (however, "priority requests" as exposed in this message have not been kept 2325 for inclusion). 2326 2327 (Contributed by Antoine Pitrou.) 2328 2329* Regular and recursive locks now accept an optional *timeout* argument to their 2330 :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou; 2331 :issue:`7316`.) 2332 2333* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout* 2334 argument. (Contributed by Torsten Landschoff; :issue:`850728`.) 2335 2336* Regular and recursive lock acquisitions can now be interrupted by signals on 2337 platforms using Pthreads. This means that Python programs that deadlock while 2338 acquiring locks can be successfully killed by repeatedly sending SIGINT to the 2339 process (by pressing :kbd:`Ctrl+C` in most shells). 2340 (Contributed by Reid Kleckner; :issue:`8844`.) 2341 2342 2343Optimizations 2344============= 2345 2346A number of small performance enhancements have been added: 2347 2348* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as 2349 being a test for membership in a set of constants. The optimizer recasts the 2350 :class:`set` as a :class:`frozenset` and stores the pre-built constant. 2351 2352 Now that the speed penalty is gone, it is practical to start writing 2353 membership tests using set-notation. This style is both semantically clear 2354 and operationally fast:: 2355 2356 extension = name.rpartition('.')[2] 2357 if extension in {'xml', 'html', 'xhtml', 'css'}: 2358 handle(name) 2359 2360 (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`). 2361 2362* Serializing and unserializing data using the :mod:`pickle` module is now 2363 several times faster. 2364 2365 (Contributed by Alexandre Vassalotti, Antoine Pitrou 2366 and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) 2367 2368* The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in 2369 :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory 2370 when called with a :term:`key function`. Previously, every element of 2371 a list was wrapped with a temporary object that remembered the key value 2372 associated with each element. Now, two arrays of keys and values are 2373 sorted in parallel. This saves the memory consumed by the sort wrappers, 2374 and it saves time lost to delegating comparisons. 2375 2376 (Patch by Daniel Stutzbach in :issue:`9915`.) 2377 2378* JSON decoding performance is improved and memory consumption is reduced 2379 whenever the same string is repeated for multiple keys. Also, JSON encoding 2380 now uses the C speedups when the ``sort_keys`` argument is true. 2381 2382 (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and 2383 Antoine Pitrou in :issue:`10314`.) 2384 2385* Recursive locks (created with the :func:`threading.RLock` API) now benefit 2386 from a C implementation which makes them as fast as regular locks, and between 2387 10x and 15x faster than their previous pure Python implementation. 2388 2389 (Contributed by Antoine Pitrou; :issue:`3001`.) 2390 2391* The fast-search algorithm in stringlib is now used by the :meth:`split`, 2392 :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on 2393 :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the 2394 algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and 2395 :meth:`rpartition`. 2396 2397 (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) 2398 2399 2400* Integer to string conversions now work two "digits" at a time, reducing the 2401 number of division and modulo operations. 2402 2403 (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.) 2404 2405There were several other minor optimizations. Set differencing now runs faster 2406when one operand is much larger than the other (patch by Andress Bennetts in 2407:issue:`8685`). The :meth:`array.repeat` method has a faster implementation 2408(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler` 2409has more efficient buffering (:issue:`3709` by Andrew Schaaf). The 2410:func:`operator.attrgetter` function has been sped-up (:issue:`10160` by 2411Christos Georgiou). And :class:`ConfigParser` loads multi-line arguments a bit 2412faster (:issue:`7113` by Łukasz Langa). 2413 2414 2415Unicode 2416======= 2417 2418Python has been updated to `Unicode 6.0.0 2419<http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds 2420over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_ 2421symbols which are important for mobile phones. 2422 2423In addition, the updated standard has altered the character properties for two 2424Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character 2425(U+19DA), making the former eligible for use in identifiers while disqualifying 2426the latter. For more information, see `Unicode Character Database Changes 2427<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_. 2428 2429 2430Codecs 2431====== 2432 2433Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`). 2434 2435MBCS encoding no longer ignores the error handler argument. In the default 2436strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an 2437undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable 2438character. 2439 2440The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for 2441decoding, and ``'strict'`` and ``'replace'`` for encoding. 2442 2443To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding 2444and the ``'replace'`` handler for encoding. 2445 2446On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than 2447the locale encoding. 2448 2449By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of 2450``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating 2451systems. 2452 2453 2454Documentation 2455============= 2456 2457The documentation continues to be improved. 2458 2459* A table of quick links has been added to the top of lengthy sections such as 2460 :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are 2461 accompanied by tables of cheatsheet-style summaries to provide an overview and 2462 memory jog without having to read all of the docs. 2463 2464* In some cases, the pure Python source code can be a helpful adjunct to the 2465 documentation, so now many modules now feature quick links to the latest 2466 version of the source code. For example, the :mod:`functools` module 2467 documentation has a quick link at the top labeled: 2468 2469 **Source code** :source:`Lib/functools.py`. 2470 2471 (Contributed by Raymond Hettinger; see 2472 `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) 2473 2474* The docs now contain more examples and recipes. In particular, :mod:`re` 2475 module has an extensive section, :ref:`re-examples`. Likewise, the 2476 :mod:`itertools` module continues to be updated with new 2477 :ref:`itertools-recipes`. 2478 2479* The :mod:`datetime` module now has an auxiliary implementation in pure Python. 2480 No functionality was changed. This just provides an easier-to-read alternate 2481 implementation. 2482 2483 (Contributed by Alexander Belopolsky in :issue:`9528`.) 2484 2485* The unmaintained :file:`Demo` directory has been removed. Some demos were 2486 integrated into the documentation, some were moved to the :file:`Tools/demo` 2487 directory, and others were removed altogether. 2488 2489 (Contributed by Georg Brandl in :issue:`7962`.) 2490 2491 2492IDLE 2493==== 2494 2495* The format menu now has an option to clean source files by stripping 2496 trailing whitespace. 2497 2498 (Contributed by Raymond Hettinger; :issue:`5150`.) 2499 2500* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk. 2501 2502 (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.) 2503 2504Code Repository 2505=============== 2506 2507In addition to the existing Subversion code repository at http://svn.python.org 2508there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at 2509https://hg.python.org/\ . 2510 2511After the 3.2 release, there are plans to switch to Mercurial as the primary 2512repository. This distributed version control system should make it easier for 2513members of the community to create and share external changesets. See 2514:pep:`385` for details. 2515 2516To learn to use the new version control system, see the `Quick Start 2517<https://www.mercurial-scm.org/wiki/QuickStart>`_ or the `Guide to 2518Mercurial Workflows <https://www.mercurial-scm.org/guide>`_. 2519 2520 2521Build and C API Changes 2522======================= 2523 2524Changes to Python's build process and to the C API include: 2525 2526* The *idle*, *pydoc* and *2to3* scripts are now installed with a 2527 version-specific suffix on ``make altinstall`` (:issue:`10679`). 2528 2529* The C functions that access the Unicode Database now accept and return 2530 characters from the full Unicode range, even on narrow unicode builds 2531 (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference 2532 in Python is that :func:`unicodedata.numeric` now returns the correct value 2533 for large code points, and :func:`repr` may consider more characters as 2534 printable. 2535 2536 (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) 2537 2538* Computed gotos are now enabled by default on supported compilers (which are 2539 detected by the configure script). They can still be disabled selectively by 2540 specifying ``--without-computed-gotos``. 2541 2542 (Contributed by Antoine Pitrou; :issue:`9203`.) 2543 2544* The option ``--with-wctype-functions`` was removed. The built-in unicode 2545 database is now used for all functions. 2546 2547 (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) 2548 2549* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is 2550 defined to be the same size as a pointer. Previously they were of type long, 2551 which on some 64-bit operating systems is still only 32 bits long. As a 2552 result of this fix, :class:`set` and :class:`dict` can now hold more than 2553 ``2**32`` entries on builds with 64-bit pointers (previously, they could grow 2554 to that size but their performance degraded catastrophically). 2555 2556 (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; 2557 :issue:`9778`.) 2558 2559* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument 2560 list. It is equivalent to C99 *va_copy* but available on all Python platforms 2561 (:issue:`2443`). 2562 2563* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter 2564 to set :attr:`sys.argv` without also modifying :attr:`sys.path` 2565 (:issue:`5753`). 2566 2567* :c:macro:`PyEval_CallObject` is now only available in macro form. The 2568 function declaration, which was kept for backwards compatibility reasons, is 2569 now removed -- the macro was introduced in 1997 (:issue:`8276`). 2570 2571* There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which 2572 is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to 2573 convert Python :class:`int` into a native fixed-width type while providing 2574 detection of cases where the conversion won't fit (:issue:`7767`). 2575 2576* The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not 2577 equal* if the Python string is *NUL* terminated. 2578 2579* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is 2580 like :c:func:`PyErr_NewException` but allows a docstring to be specified. 2581 This lets C exceptions have the same self-documenting capabilities as 2582 their pure Python counterparts (:issue:`7033`). 2583 2584* When compiled with the ``--with-valgrind`` option, the pymalloc 2585 allocator will be automatically disabled when running under Valgrind. This 2586 gives improved memory leak detection when running under Valgrind, while taking 2587 advantage of pymalloc at other times (:issue:`2422`). 2588 2589* Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no 2590 longer used and it had never been documented (:issue:`8837`). 2591 2592There were a number of other small changes to the C-API. See the 2593:source:`Misc/NEWS` file for a complete list. 2594 2595Also, there were a number of updates to the Mac OS X build, see 2596:source:`Mac/BuildScript/README.txt` for details. For users running a 32/64-bit 2597build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. 2598Accordingly, we recommend installing an updated alternative such as 2599`ActiveState Tcl/Tk 8.5.9 <https://www.activestate.com/activetcl/downloads>`_\. 2600See https://www.python.org/download/mac/tcltk/ for additional details. 2601 2602Porting to Python 3.2 2603===================== 2604 2605This section lists previously described changes and other bugfixes that may 2606require changes to your code: 2607 2608* The :mod:`configparser` module has a number of clean-ups. The major change is 2609 to replace the old :class:`ConfigParser` class with long-standing preferred 2610 alternative :class:`SafeConfigParser`. In addition there are a number of 2611 smaller incompatibilities: 2612 2613 * The interpolation syntax is now validated on 2614 :meth:`~configparser.ConfigParser.get` and 2615 :meth:`~configparser.ConfigParser.set` operations. In the default 2616 interpolation scheme, only two tokens with percent signs are valid: ``%(name)s`` 2617 and ``%%``, the latter being an escaped percent sign. 2618 2619 * The :meth:`~configparser.ConfigParser.set` and 2620 :meth:`~configparser.ConfigParser.add_section` methods now verify that 2621 values are actual strings. Formerly, unsupported types could be introduced 2622 unintentionally. 2623 2624 * Duplicate sections or options from a single source now raise either 2625 :exc:`~configparser.DuplicateSectionError` or 2626 :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would 2627 silently overwrite a previous entry. 2628 2629 * Inline comments are now disabled by default so now the **;** character 2630 can be safely used in values. 2631 2632 * Comments now can be indented. Consequently, for **;** or **#** to appear at 2633 the start of a line in multiline values, it has to be interpolated. This 2634 keeps comment prefix characters in values from being mistaken as comments. 2635 2636 * ``""`` is now a valid value and is no longer automatically converted to an 2637 empty string. For empty strings, use ``"option ="`` in a line. 2638 2639* The :mod:`nntplib` module was reworked extensively, meaning that its APIs 2640 are often incompatible with the 3.1 APIs. 2641 2642* :class:`bytearray` objects can no longer be used as filenames; instead, 2643 they should be converted to :class:`bytes`. 2644 2645* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to 2646 :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names 2647 have been deprecated. (See :issue:`8990`.) 2648 2649* ``PyArg_Parse*()`` functions: 2650 2651 * "t#" format has been removed: use "s#" or "s*" instead 2652 * "w" and "w#" formats has been removed: use "w*" instead 2653 2654* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap 2655 opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used 2656 instead; the new type has a well-defined interface for passing typing safety 2657 information and a less complicated signature for calling a destructor. 2658 2659* The :func:`sys.setfilesystemencoding` function was removed because 2660 it had a flawed design. 2661 2662* The :func:`random.seed` function and method now salt string seeds with an 2663 sha512 hash function. To access the previous version of *seed* in order to 2664 reproduce Python 3.1 sequences, set the *version* argument to *1*, 2665 ``random.seed(s, version=1)``. 2666 2667* The previously deprecated :func:`string.maketrans` function has been removed 2668 in favor of the static methods :meth:`bytes.maketrans` and 2669 :meth:`bytearray.maketrans`. This change solves the confusion around which 2670 types were supported by the :mod:`string` module. Now, :class:`str`, 2671 :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and 2672 **translate** methods with intermediate translation tables of the appropriate 2673 type. 2674 2675 (Contributed by Georg Brandl; :issue:`5675`.) 2676 2677* The previously deprecated :func:`contextlib.nested` function has been removed 2678 in favor of a plain :keyword:`with` statement which can accept multiple 2679 context managers. The latter technique is faster (because it is built-in), 2680 and it does a better job finalizing multiple context managers when one of them 2681 raises an exception:: 2682 2683 with open('mylog.txt') as infile, open('a.out', 'w') as outfile: 2684 for line in infile: 2685 if '<critical>' in line: 2686 outfile.write(line) 2687 2688 (Contributed by Georg Brandl and Mattias Brändström; 2689 `appspot issue 53094 <https://codereview.appspot.com/53094>`_.) 2690 2691* :func:`struct.pack` now only allows bytes for the ``s`` string pack code. 2692 Formerly, it would accept text arguments and implicitly encode them to bytes 2693 using UTF-8. This was problematic because it made assumptions about the 2694 correct encoding and because a variable-length encoding can fail when writing 2695 to fixed length segment of a structure. 2696 2697 Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten 2698 with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``. 2699 2700 (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.) 2701 2702* The :class:`xml.etree.ElementTree` class now raises an 2703 :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it 2704 raised an :exc:`xml.parsers.expat.ExpatError`. 2705 2706* The new, longer :func:`str` value on floats may break doctests which rely on 2707 the old output format. 2708 2709* In :class:`subprocess.Popen`, the default value for *close_fds* is now 2710 ``True`` under Unix; under Windows, it is ``True`` if the three standard 2711 streams are set to ``None``, ``False`` otherwise. Previously, *close_fds* 2712 was always ``False`` by default, which produced difficult to solve bugs 2713 or race conditions when open file descriptors would leak into the child 2714 process. 2715 2716* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request` 2717 and :mod:`http.client`. Such support is still present on the server side 2718 (in :mod:`http.server`). 2719 2720 (Contributed by Antoine Pitrou, :issue:`10711`.) 2721 2722* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout 2723 occurs, rather than a generic :exc:`~ssl.SSLError`. 2724 2725 (Contributed by Antoine Pitrou, :issue:`10272`.) 2726 2727* The misleading functions :c:func:`PyEval_AcquireLock()` and 2728 :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The 2729 thread-state aware APIs (such as :c:func:`PyEval_SaveThread()` 2730 and :c:func:`PyEval_RestoreThread()`) should be used instead. 2731 2732* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and 2733 a new function, :func:`asyncore.handle_accepted`, was added to replace it. 2734 2735 (Contributed by Giampaolo Rodola in :issue:`6706`.) 2736 2737* Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` 2738 cannot be called before :c:func:`Py_Initialize()` anymore. 2739