1:mod:`email.charset`: Representing character sets
2-------------------------------------------------
3
4.. module:: email.charset
5   :synopsis: Character Sets
6
7**Source code:** :source:`Lib/email/charset.py`
8
9--------------
10
11This module is part of the legacy (``Compat32``) email API.  In the new
12API only the aliases table is used.
13
14The remaining text in this section is the original documentation of the module.
15
16This module provides a class :class:`Charset` for representing character sets
17and character set conversions in email messages, as well as a character set
18registry and several convenience methods for manipulating this registry.
19Instances of :class:`Charset` are used in several other modules within the
20:mod:`email` package.
21
22Import this class from the :mod:`email.charset` module.
23
24
25.. class:: Charset(input_charset=DEFAULT_CHARSET)
26
27   Map character sets to their email properties.
28
29   This class provides information about the requirements imposed on email for a
30   specific character set.  It also provides convenience routines for converting
31   between character sets, given the availability of the applicable codecs.  Given
32   a character set, it will do its best to provide information on how to use that
33   character set in an email message in an RFC-compliant way.
34
35   Certain character sets must be encoded with quoted-printable or base64 when used
36   in email headers or bodies.  Certain character sets must be converted outright,
37   and are not allowed in email.
38
39   Optional *input_charset* is as described below; it is always coerced to lower
40   case.  After being alias normalized it is also used as a lookup into the
41   registry of character sets to find out the header encoding, body encoding, and
42   output conversion codec to be used for the character set.  For example, if
43   *input_charset* is ``iso-8859-1``, then headers and bodies will be encoded using
44   quoted-printable and no output conversion codec is necessary.  If
45   *input_charset* is ``euc-jp``, then headers will be encoded with base64, bodies
46   will not be encoded, but output text will be converted from the ``euc-jp``
47   character set to the ``iso-2022-jp`` character set.
48
49   :class:`Charset` instances have the following data attributes:
50
51   .. attribute:: input_charset
52
53      The initial character set specified.  Common aliases are converted to
54      their *official* email names (e.g. ``latin_1`` is converted to
55      ``iso-8859-1``).  Defaults to 7-bit ``us-ascii``.
56
57
58   .. attribute:: header_encoding
59
60      If the character set must be encoded before it can be used in an email
61      header, this attribute will be set to ``Charset.QP`` (for
62      quoted-printable), ``Charset.BASE64`` (for base64 encoding), or
63      ``Charset.SHORTEST`` for the shortest of QP or BASE64 encoding. Otherwise,
64      it will be ``None``.
65
66
67   .. attribute:: body_encoding
68
69      Same as *header_encoding*, but describes the encoding for the mail
70      message's body, which indeed may be different than the header encoding.
71      ``Charset.SHORTEST`` is not allowed for *body_encoding*.
72
73
74   .. attribute:: output_charset
75
76      Some character sets must be converted before they can be used in email
77      headers or bodies.  If the *input_charset* is one of them, this attribute
78      will contain the name of the character set output will be converted to.
79      Otherwise, it will be ``None``.
80
81
82   .. attribute:: input_codec
83
84      The name of the Python codec used to convert the *input_charset* to
85      Unicode.  If no conversion codec is necessary, this attribute will be
86      ``None``.
87
88
89   .. attribute:: output_codec
90
91      The name of the Python codec used to convert Unicode to the
92      *output_charset*.  If no conversion codec is necessary, this attribute
93      will have the same value as the *input_codec*.
94
95
96   :class:`Charset` instances also have the following methods:
97
98   .. method:: get_body_encoding()
99
100      Return the content transfer encoding used for body encoding.
101
102      This is either the string ``quoted-printable`` or ``base64`` depending on
103      the encoding used, or it is a function, in which case you should call the
104      function with a single argument, the Message object being encoded.  The
105      function should then set the :mailheader:`Content-Transfer-Encoding`
106      header itself to whatever is appropriate.
107
108      Returns the string ``quoted-printable`` if *body_encoding* is ``QP``,
109      returns the string ``base64`` if *body_encoding* is ``BASE64``, and
110      returns the string ``7bit`` otherwise.
111
112
113   .. XXX to_splittable and from_splittable are not there anymore!
114
115   .. to_splittable(s)
116
117      Convert a possibly multibyte string to a safely splittable format. *s* is
118      the string to split.
119
120      Uses the *input_codec* to try and convert the string to Unicode, so it can
121      be safely split on character boundaries (even for multibyte characters).
122
123      Returns the string as-is if it isn't known how to convert *s* to Unicode
124      with the *input_charset*.
125
126      Characters that could not be converted to Unicode will be replaced with
127      the Unicode replacement character ``'U+FFFD'``.
128
129
130   .. from_splittable(ustr[, to_output])
131
132      Convert a splittable string back into an encoded string.  *ustr* is a
133      Unicode string to "unsplit".
134
135      This method uses the proper codec to try and convert the string from
136      Unicode back into an encoded format.  Return the string as-is if it is not
137      Unicode, or if it could not be converted from Unicode.
138
139      Characters that could not be converted from Unicode will be replaced with
140      an appropriate character (usually ``'?'``).
141
142      If *to_output* is ``True`` (the default), uses *output_codec* to convert
143      to an encoded format.  If *to_output* is ``False``, it uses *input_codec*.
144
145
146   .. method:: get_output_charset()
147
148      Return the output character set.
149
150      This is the *output_charset* attribute if that is not ``None``, otherwise
151      it is *input_charset*.
152
153
154   .. method:: header_encode(string)
155
156      Header-encode the string *string*.
157
158      The type of encoding (base64 or quoted-printable) will be based on the
159      *header_encoding* attribute.
160
161
162   .. method:: header_encode_lines(string, maxlengths)
163
164      Header-encode a *string* by converting it first to bytes.
165
166      This is similar to :meth:`header_encode` except that the string is fit
167      into maximum line lengths as given by the argument *maxlengths*, which
168      must be an iterator: each element returned from this iterator will provide
169      the next maximum line length.
170
171
172   .. method:: body_encode(string)
173
174      Body-encode the string *string*.
175
176      The type of encoding (base64 or quoted-printable) will be based on the
177      *body_encoding* attribute.
178
179   The :class:`Charset` class also provides a number of methods to support
180   standard operations and built-in functions.
181
182
183   .. method:: __str__()
184
185      Returns *input_charset* as a string coerced to lower
186      case. :meth:`__repr__` is an alias for :meth:`__str__`.
187
188
189   .. method:: __eq__(other)
190
191      This method allows you to compare two :class:`Charset` instances for
192      equality.
193
194
195   .. method:: __ne__(other)
196
197      This method allows you to compare two :class:`Charset` instances for
198      inequality.
199
200The :mod:`email.charset` module also provides the following functions for adding
201new entries to the global character set, alias, and codec registries:
202
203
204.. function:: add_charset(charset, header_enc=None, body_enc=None, output_charset=None)
205
206   Add character properties to the global registry.
207
208   *charset* is the input character set, and must be the canonical name of a
209   character set.
210
211   Optional *header_enc* and *body_enc* is either ``Charset.QP`` for
212   quoted-printable, ``Charset.BASE64`` for base64 encoding,
213   ``Charset.SHORTEST`` for the shortest of quoted-printable or base64 encoding,
214   or ``None`` for no encoding.  ``SHORTEST`` is only valid for
215   *header_enc*. The default is ``None`` for no encoding.
216
217   Optional *output_charset* is the character set that the output should be in.
218   Conversions will proceed from input charset, to Unicode, to the output charset
219   when the method :meth:`Charset.convert` is called.  The default is to output in
220   the same character set as the input.
221
222   Both *input_charset* and *output_charset* must have Unicode codec entries in the
223   module's character set-to-codec mapping; use :func:`add_codec` to add codecs the
224   module does not know about.  See the :mod:`codecs` module's documentation for
225   more information.
226
227   The global character set registry is kept in the module global dictionary
228   ``CHARSETS``.
229
230
231.. function:: add_alias(alias, canonical)
232
233   Add a character set alias.  *alias* is the alias name, e.g. ``latin-1``.
234   *canonical* is the character set's canonical name, e.g. ``iso-8859-1``.
235
236   The global charset alias registry is kept in the module global dictionary
237   ``ALIASES``.
238
239
240.. function:: add_codec(charset, codecname)
241
242   Add a codec that map characters in the given character set to and from Unicode.
243
244   *charset* is the canonical name of a character set. *codecname* is the name of a
245   Python codec, as appropriate for the second argument to the :class:`str`'s
246   :meth:`~str.encode` method.
247
248