1.. _tut-brieftourtwo:
2
3**********************************************
4Brief Tour of the Standard Library --- Part II
5**********************************************
6
7This second tour covers more advanced modules that support professional
8programming needs.  These modules rarely occur in small scripts.
9
10
11.. _tut-output-formatting:
12
13Output Formatting
14=================
15
16The :mod:`reprlib` module provides a version of :func:`repr` customized for
17abbreviated displays of large or deeply nested containers::
18
19   >>> import reprlib
20   >>> reprlib.repr(set('supercalifragilisticexpialidocious'))
21   "{'a', 'c', 'd', 'e', 'f', 'g', ...}"
22
23The :mod:`pprint` module offers more sophisticated control over printing both
24built-in and user defined objects in a way that is readable by the interpreter.
25When the result is longer than one line, the "pretty printer" adds line breaks
26and indentation to more clearly reveal data structure::
27
28   >>> import pprint
29   >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
30   ...     'yellow'], 'blue']]]
31   ...
32   >>> pprint.pprint(t, width=30)
33   [[[['black', 'cyan'],
34      'white',
35      ['green', 'red']],
36     [['magenta', 'yellow'],
37      'blue']]]
38
39The :mod:`textwrap` module formats paragraphs of text to fit a given screen
40width::
41
42   >>> import textwrap
43   >>> doc = """The wrap() method is just like fill() except that it returns
44   ... a list of strings instead of one big string with newlines to separate
45   ... the wrapped lines."""
46   ...
47   >>> print(textwrap.fill(doc, width=40))
48   The wrap() method is just like fill()
49   except that it returns a list of strings
50   instead of one big string with newlines
51   to separate the wrapped lines.
52
53The :mod:`locale` module accesses a database of culture specific data formats.
54The grouping attribute of locale's format function provides a direct way of
55formatting numbers with group separators::
56
57   >>> import locale
58   >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
59   'English_United States.1252'
60   >>> conv = locale.localeconv()          # get a mapping of conventions
61   >>> x = 1234567.8
62   >>> locale.format("%d", x, grouping=True)
63   '1,234,567'
64   >>> locale.format_string("%s%.*f", (conv['currency_symbol'],
65   ...                      conv['frac_digits'], x), grouping=True)
66   '$1,234,567.80'
67
68
69.. _tut-templating:
70
71Templating
72==========
73
74The :mod:`string` module includes a versatile :class:`~string.Template` class
75with a simplified syntax suitable for editing by end-users.  This allows users
76to customize their applications without having to alter the application.
77
78The format uses placeholder names formed by ``$`` with valid Python identifiers
79(alphanumeric characters and underscores).  Surrounding the placeholder with
80braces allows it to be followed by more alphanumeric letters with no intervening
81spaces.  Writing ``$$`` creates a single escaped ``$``::
82
83   >>> from string import Template
84   >>> t = Template('${village}folk send $$10 to $cause.')
85   >>> t.substitute(village='Nottingham', cause='the ditch fund')
86   'Nottinghamfolk send $10 to the ditch fund.'
87
88The :meth:`~string.Template.substitute` method raises a :exc:`KeyError` when a
89placeholder is not supplied in a dictionary or a keyword argument.  For
90mail-merge style applications, user supplied data may be incomplete and the
91:meth:`~string.Template.safe_substitute` method may be more appropriate ---
92it will leave placeholders unchanged if data is missing::
93
94   >>> t = Template('Return the $item to $owner.')
95   >>> d = dict(item='unladen swallow')
96   >>> t.substitute(d)
97   Traceback (most recent call last):
98     ...
99   KeyError: 'owner'
100   >>> t.safe_substitute(d)
101   'Return the unladen swallow to $owner.'
102
103Template subclasses can specify a custom delimiter.  For example, a batch
104renaming utility for a photo browser may elect to use percent signs for
105placeholders such as the current date, image sequence number, or file format::
106
107   >>> import time, os.path
108   >>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
109   >>> class BatchRename(Template):
110   ...     delimiter = '%'
111   >>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format):  ')
112   Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f
113
114   >>> t = BatchRename(fmt)
115   >>> date = time.strftime('%d%b%y')
116   >>> for i, filename in enumerate(photofiles):
117   ...     base, ext = os.path.splitext(filename)
118   ...     newname = t.substitute(d=date, n=i, f=ext)
119   ...     print('{0} --> {1}'.format(filename, newname))
120
121   img_1074.jpg --> Ashley_0.jpg
122   img_1076.jpg --> Ashley_1.jpg
123   img_1077.jpg --> Ashley_2.jpg
124
125Another application for templating is separating program logic from the details
126of multiple output formats.  This makes it possible to substitute custom
127templates for XML files, plain text reports, and HTML web reports.
128
129
130.. _tut-binary-formats:
131
132Working with Binary Data Record Layouts
133=======================================
134
135The :mod:`struct` module provides :func:`~struct.pack` and
136:func:`~struct.unpack` functions for working with variable length binary
137record formats.  The following example shows
138how to loop through header information in a ZIP file without using the
139:mod:`zipfile` module.  Pack codes ``"H"`` and ``"I"`` represent two and four
140byte unsigned numbers respectively.  The ``"<"`` indicates that they are
141standard size and in little-endian byte order::
142
143   import struct
144
145   with open('myfile.zip', 'rb') as f:
146       data = f.read()
147
148   start = 0
149   for i in range(3):                      # show the first 3 file headers
150       start += 14
151       fields = struct.unpack('<IIIHH', data[start:start+16])
152       crc32, comp_size, uncomp_size, filenamesize, extra_size = fields
153
154       start += 16
155       filename = data[start:start+filenamesize]
156       start += filenamesize
157       extra = data[start:start+extra_size]
158       print(filename, hex(crc32), comp_size, uncomp_size)
159
160       start += extra_size + comp_size     # skip to the next header
161
162
163.. _tut-multi-threading:
164
165Multi-threading
166===============
167
168Threading is a technique for decoupling tasks which are not sequentially
169dependent.  Threads can be used to improve the responsiveness of applications
170that accept user input while other tasks run in the background.  A related use
171case is running I/O in parallel with computations in another thread.
172
173The following code shows how the high level :mod:`threading` module can run
174tasks in background while the main program continues to run::
175
176   import threading, zipfile
177
178   class AsyncZip(threading.Thread):
179       def __init__(self, infile, outfile):
180           threading.Thread.__init__(self)
181           self.infile = infile
182           self.outfile = outfile
183
184       def run(self):
185           f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
186           f.write(self.infile)
187           f.close()
188           print('Finished background zip of:', self.infile)
189
190   background = AsyncZip('mydata.txt', 'myarchive.zip')
191   background.start()
192   print('The main program continues to run in foreground.')
193
194   background.join()    # Wait for the background task to finish
195   print('Main program waited until background was done.')
196
197The principal challenge of multi-threaded applications is coordinating threads
198that share data or other resources.  To that end, the threading module provides
199a number of synchronization primitives including locks, events, condition
200variables, and semaphores.
201
202While those tools are powerful, minor design errors can result in problems that
203are difficult to reproduce.  So, the preferred approach to task coordination is
204to concentrate all access to a resource in a single thread and then use the
205:mod:`queue` module to feed that thread with requests from other threads.
206Applications using :class:`~queue.Queue` objects for inter-thread communication and
207coordination are easier to design, more readable, and more reliable.
208
209
210.. _tut-logging:
211
212Logging
213=======
214
215The :mod:`logging` module offers a full featured and flexible logging system.
216At its simplest, log messages are sent to a file or to ``sys.stderr``::
217
218   import logging
219   logging.debug('Debugging information')
220   logging.info('Informational message')
221   logging.warning('Warning:config file %s not found', 'server.conf')
222   logging.error('Error occurred')
223   logging.critical('Critical error -- shutting down')
224
225This produces the following output:
226
227.. code-block:: none
228
229   WARNING:root:Warning:config file server.conf not found
230   ERROR:root:Error occurred
231   CRITICAL:root:Critical error -- shutting down
232
233By default, informational and debugging messages are suppressed and the output
234is sent to standard error.  Other output options include routing messages
235through email, datagrams, sockets, or to an HTTP Server.  New filters can select
236different routing based on message priority: :const:`~logging.DEBUG`,
237:const:`~logging.INFO`, :const:`~logging.WARNING`, :const:`~logging.ERROR`,
238and :const:`~logging.CRITICAL`.
239
240The logging system can be configured directly from Python or can be loaded from
241a user editable configuration file for customized logging without altering the
242application.
243
244
245.. _tut-weak-references:
246
247Weak References
248===============
249
250Python does automatic memory management (reference counting for most objects and
251:term:`garbage collection` to eliminate cycles).  The memory is freed shortly
252after the last reference to it has been eliminated.
253
254This approach works fine for most applications but occasionally there is a need
255to track objects only as long as they are being used by something else.
256Unfortunately, just tracking them creates a reference that makes them permanent.
257The :mod:`weakref` module provides tools for tracking objects without creating a
258reference.  When the object is no longer needed, it is automatically removed
259from a weakref table and a callback is triggered for weakref objects.  Typical
260applications include caching objects that are expensive to create::
261
262   >>> import weakref, gc
263   >>> class A:
264   ...     def __init__(self, value):
265   ...         self.value = value
266   ...     def __repr__(self):
267   ...         return str(self.value)
268   ...
269   >>> a = A(10)                   # create a reference
270   >>> d = weakref.WeakValueDictionary()
271   >>> d['primary'] = a            # does not create a reference
272   >>> d['primary']                # fetch the object if it is still alive
273   10
274   >>> del a                       # remove the one reference
275   >>> gc.collect()                # run garbage collection right away
276   0
277   >>> d['primary']                # entry was automatically removed
278   Traceback (most recent call last):
279     File "<stdin>", line 1, in <module>
280       d['primary']                # entry was automatically removed
281     File "C:/python38/lib/weakref.py", line 46, in __getitem__
282       o = self.data[key]()
283   KeyError: 'primary'
284
285
286.. _tut-list-tools:
287
288Tools for Working with Lists
289============================
290
291Many data structure needs can be met with the built-in list type. However,
292sometimes there is a need for alternative implementations with different
293performance trade-offs.
294
295The :mod:`array` module provides an :class:`~array.array()` object that is like
296a list that stores only homogeneous data and stores it more compactly.  The
297following example shows an array of numbers stored as two byte unsigned binary
298numbers (typecode ``"H"``) rather than the usual 16 bytes per entry for regular
299lists of Python int objects::
300
301   >>> from array import array
302   >>> a = array('H', [4000, 10, 700, 22222])
303   >>> sum(a)
304   26932
305   >>> a[1:3]
306   array('H', [10, 700])
307
308The :mod:`collections` module provides a :class:`~collections.deque()` object
309that is like a list with faster appends and pops from the left side but slower
310lookups in the middle. These objects are well suited for implementing queues
311and breadth first tree searches::
312
313   >>> from collections import deque
314   >>> d = deque(["task1", "task2", "task3"])
315   >>> d.append("task4")
316   >>> print("Handling", d.popleft())
317   Handling task1
318
319::
320
321   unsearched = deque([starting_node])
322   def breadth_first_search(unsearched):
323       node = unsearched.popleft()
324       for m in gen_moves(node):
325           if is_goal(m):
326               return m
327           unsearched.append(m)
328
329In addition to alternative list implementations, the library also offers other
330tools such as the :mod:`bisect` module with functions for manipulating sorted
331lists::
332
333   >>> import bisect
334   >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
335   >>> bisect.insort(scores, (300, 'ruby'))
336   >>> scores
337   [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]
338
339The :mod:`heapq` module provides functions for implementing heaps based on
340regular lists.  The lowest valued entry is always kept at position zero.  This
341is useful for applications which repeatedly access the smallest element but do
342not want to run a full list sort::
343
344   >>> from heapq import heapify, heappop, heappush
345   >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
346   >>> heapify(data)                      # rearrange the list into heap order
347   >>> heappush(data, -5)                 # add a new entry
348   >>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
349   [-5, 0, 1]
350
351
352.. _tut-decimal-fp:
353
354Decimal Floating Point Arithmetic
355=================================
356
357The :mod:`decimal` module offers a :class:`~decimal.Decimal` datatype for
358decimal floating point arithmetic.  Compared to the built-in :class:`float`
359implementation of binary floating point, the class is especially helpful for
360
361* financial applications and other uses which require exact decimal
362  representation,
363* control over precision,
364* control over rounding to meet legal or regulatory requirements,
365* tracking of significant decimal places, or
366* applications where the user expects the results to match calculations done by
367  hand.
368
369For example, calculating a 5% tax on a 70 cent phone charge gives different
370results in decimal floating point and binary floating point. The difference
371becomes significant if the results are rounded to the nearest cent::
372
373   >>> from decimal import *
374   >>> round(Decimal('0.70') * Decimal('1.05'), 2)
375   Decimal('0.74')
376   >>> round(.70 * 1.05, 2)
377   0.73
378
379The :class:`~decimal.Decimal` result keeps a trailing zero, automatically
380inferring four place significance from multiplicands with two place
381significance.  Decimal reproduces mathematics as done by hand and avoids
382issues that can arise when binary floating point cannot exactly represent
383decimal quantities.
384
385Exact representation enables the :class:`~decimal.Decimal` class to perform
386modulo calculations and equality tests that are unsuitable for binary floating
387point::
388
389   >>> Decimal('1.00') % Decimal('.10')
390   Decimal('0.00')
391   >>> 1.00 % 0.10
392   0.09999999999999995
393
394   >>> sum([Decimal('0.1')]*10) == Decimal('1.0')
395   True
396   >>> sum([0.1]*10) == 1.0
397   False
398
399The :mod:`decimal` module provides arithmetic with as much precision as needed::
400
401   >>> getcontext().prec = 36
402   >>> Decimal(1) / Decimal(7)
403   Decimal('0.142857142857142857142857142857142857')
404
405
406