1# -*- mode: org -*- 2#+TITLE: GNU Privacy Guard (GnuPG) Made Easy Python Bindings HOWTO (English) 3#+AUTHOR: Ben McGinnes 4#+LATEX_COMPILER: xelatex 5#+LATEX_CLASS: article 6#+LATEX_CLASS_OPTIONS: [12pt] 7#+LATEX_HEADER: \usepackage{xltxtra} 8#+LATEX_HEADER: \usepackage[margin=1in]{geometry} 9#+LATEX_HEADER: \setmainfont[Ligatures={Common}]{Times New Roman} 10#+LATEX_HEADER: \author{Ben McGinnes <ben@gnupg.org>} 11 12 13* Introduction 14 :PROPERTIES: 15 :CUSTOM_ID: intro 16 :END: 17 18| Version: | 0.1.5 | 19| GPGME Version: | 1.13.0 | 20| Author: | Ben McGinnes <ben@gnupg.org> | 21| Author GPG Key: | DB4724E6FA4286C92B4E55C4321E4E2373590E5D | 22| Language: | Australian English, British English | 23| Language codes: | en-AU, en-GB, en | 24 25This document provides basic instruction in how to use the GPGME 26Python bindings to programmatically leverage the GPGME library. 27 28 29** Python 2 versus Python 3 30 :PROPERTIES: 31 :CUSTOM_ID: py2-vs-py3 32 :END: 33 34Though the GPGME Python bindings themselves provide support for both 35Python 2 and 3, the focus is unequivocally on Python 3 and 36specifically from Python 3.4 and above. As a consequence all the 37examples and instructions in this guide use Python 3 code. 38 39Much of it will work with Python 2, but much of it also deals with 40Python 3 byte literals, particularly when reading and writing data. 41Developers concentrating on Python 2.7, and possibly even 2.6, will 42need to make the appropriate modifications to support the older string 43and unicode types as opposed to bytes. 44 45There are multiple reasons for concentrating on Python 3; some of 46which relate to the immediate integration of these bindings, some of 47which relate to longer term plans for both GPGME and the python 48bindings and some of which relate to the impending EOL period for 49Python 2.7. Essentially, though, there is little value in tying the 50bindings to a version of the language which is a dead end and the 51advantages offered by Python 3 over Python 2 make handling the data 52types with which GPGME deals considerably easier. 53 54 55** Examples 56 :PROPERTIES: 57 :CUSTOM_ID: howto-python3-examples 58 :END: 59 60All of the examples found in this document can be found as Python 3 61scripts in the =lang/python/examples/howto= directory. 62 63 64** Unofficial Drafts 65 :PROPERTIES: 66 :CUSTOM_ID: unofficial-drafts 67 :END: 68 69In addition to shipping with each release of GPGME, there is a section 70on locations to read or download [[#draft-editions][draft editions]] of this document from 71at the end of it. These are unofficial versions produced in between 72major releases. 73 74 75** What's New 76 :PROPERTIES: 77 :CUSTOM_ID: new-stuff 78 :END: 79 80Full details of what is new are now available in the [[file:what-is-new.org][What's New]] file 81and archives of the preceding /What's New/ sections are available in 82the [[file:what-was-new][What Was New]] file. 83 84 85*** New in GPGME 1·13·0 86 :PROPERTIES: 87 :CUSTOM_ID: new-stuff-1-13-0 88 :END: 89 90See the [[file:what-is-new#new-stuff-1-13-0][What's New]] document for what is new in version 1.13.0. 91 92 93*** New in GPGME 1·12·0 94 :PROPERTIES: 95 :CUSTOM_ID: new-stuff-1-12-0 96 :END: 97 98See the [[file:what-was-new#new-stuff-1-12-0][What Was New]] document for what was new in version 1.12.0. 99 100 101* GPGME Concepts 102 :PROPERTIES: 103 :CUSTOM_ID: gpgme-concepts 104 :END: 105 106 107** A C API 108 :PROPERTIES: 109 :CUSTOM_ID: gpgme-c-api 110 :END: 111 112Unlike many modern APIs with which programmers will be more familiar 113with these days, the GPGME API is a C API. The API is intended for 114use by C coders who would be able to access its features by including 115the =gpgme.h= header file with their own C source code and then access 116its functions just as they would any other C headers. 117 118This is a very effective method of gaining complete access to the API 119and in the most efficient manner possible. It does, however, have the 120drawback that it cannot be directly used by other languages without 121some means of providing an interface to those languages. This is 122where the need for bindings in various languages stems. 123 124 125** Python bindings 126 :PROPERTIES: 127 :CUSTOM_ID: gpgme-python-bindings 128 :END: 129 130The Python bindings for GPGME provide a higher level means of 131accessing the complete feature set of GPGME itself. It also provides 132a more pythonic means of calling these API functions. 133 134The bindings are generated dynamically with SWIG and the copy of 135=gpgme.h= generated when GPGME is compiled. 136 137This means that a version of the Python bindings is fundamentally tied 138to the exact same version of GPGME used to generate that copy of 139=gpgme.h=. 140 141 142** Difference between the Python bindings and other GnuPG Python packages 143 :PROPERTIES: 144 :CUSTOM_ID: gpgme-python-bindings-diffs 145 :END: 146 147There have been numerous attempts to add GnuPG support to Python over 148the years. Some of the most well known are listed here, along with 149what differentiates them. 150 151 152*** The python-gnupg package maintained by Vinay Sajip 153 :PROPERTIES: 154 :CUSTOM_ID: diffs-python-gnupg 155 :END: 156 157This is arguably the most popular means of integrating GPG with 158Python. The package utilises the =subprocess= module to implement 159wrappers for the =gpg= and =gpg2= executables normally invoked on the 160command line (=gpg.exe= and =gpg2.exe= on Windows). 161 162The popularity of this package stemmed from its ease of use and 163capability in providing the most commonly required features. 164 165Unfortunately it has been beset by a number of security issues in the 166past; most of which stemmed from using unsafe methods of accessing the 167command line via the =subprocess= calls. While some effort has been 168made over the last two to three years (as of 2018) to mitigate this, 169particularly by no longer providing shell access through those 170subprocess calls, the wrapper is still somewhat limited in the scope 171of its GnuPG features coverage. 172 173The python-gnupg package is available under the MIT license. 174 175 176*** The gnupg package created and maintained by Isis Lovecruft 177 :PROPERTIES: 178 :CUSTOM_ID: diffs-isis-gnupg 179 :END: 180 181In 2015 Isis Lovecruft from the Tor Project forked and then 182re-implemented the python-gnupg package as just gnupg. This new 183package also relied on subprocess to call the =gpg= or =gpg2= 184binaries, but did so somewhat more securely. 185 186The naming and version numbering selected for this package, however, 187resulted in conflicts with the original python-gnupg and since its 188functions were called in a different manner to python-gnupg, the 189release of this package also resulted in a great deal of consternation 190when people installed what they thought was an upgrade that 191subsequently broke the code relying on it. 192 193The gnupg package is available under the GNU General Public License 194version 3.0 (or any later version). 195 196 197*** The PyME package maintained by Martin Albrecht 198 :PROPERTIES: 199 :CUSTOM_ID: diffs-pyme 200 :END: 201 202This package is the origin of these bindings, though they are somewhat 203different now. For details of when and how the PyME package was 204folded back into GPGME itself see the [[file:short-history.org][Short History]] document.[fn:1] 205 206The PyME package was first released in 2002 and was also the first 207attempt to implement a low level binding to GPGME. In doing so it 208provided access to considerably more functionality than either the 209=python-gnupg= or =gnupg= packages. 210 211The PyME package is only available for Python 2.6 and 2.7. 212 213Porting the PyME package to Python 3.4 in 2015 is what resulted in it 214being folded into the GPGME project and the current bindings are the 215end result of that effort. 216 217The PyME package is available under the same dual licensing as GPGME 218itself: the GNU General Public License version 2.0 (or any later 219version) and the GNU Lesser General Public License version 2.1 (or any 220later version). 221 222 223* GPGME Python bindings installation 224 :PROPERTIES: 225 :CUSTOM_ID: gpgme-python-install 226 :END: 227 228 229** No PyPI 230 :PROPERTIES: 231 :CUSTOM_ID: do-not-use-pypi 232 :END: 233 234Most third-party Python packages and modules are available and 235distributed through the Python Package Installer, known as PyPI. 236 237Due to the nature of what these bindings are and how they work, it is 238infeasible to install the GPGME Python bindings in the same way. 239 240This is because the bindings use SWIG to dynamically generate C 241bindings against =gpgme.h= and =gpgme.h= is generated from 242=gpgme.h.in= at compile time when GPGME is built from source. Thus to 243include a package in PyPI which actually built correctly would require 244either statically built libraries for every architecture bundled with 245it or a full implementation of C for each architecture. 246 247See the additional notes regarding [[#snafu-cffi][CFFI and SWIG]] at the end of this 248section for further details. 249 250 251** Requirements 252 :PROPERTIES: 253 :CUSTOM_ID: gpgme-python-requirements 254 :END: 255 256The GPGME Python bindings only have three requirements: 257 2581. A suitable version of Python 2 or Python 3. With Python 2 that 259 means CPython 2.7 and with Python 3 that means CPython 3.4 or 260 higher. 2612. [[https://www.swig.org][SWIG]]. 2623. GPGME itself. Which also means that all of GPGME's dependencies 263 must be installed too. 264 265 266*** Recommended Additions 267 :PROPERTIES: 268 :CUSTOM_ID: gpgme-python-recommendations 269 :END: 270 271Though none of the following are absolute requirements, they are all 272recommended for use with the Python bindings. In some cases these 273recommendations refer to which version(s) of CPython to use the 274bindings with, while others refer to third party modules which provide 275a significant advantage in some way. 276 2771. If possible, use Python 3 instead of 2. 2782. Favour a more recent version of Python since even 3.4 is due to 279 reach EOL soon. In production systems and services, Python 3.6 280 should be robust enough to be relied on. 2813. If possible add the following Python modules which are not part of 282 the standard library: [[http://docs.python-requests.org/en/latest/index.html][Requests]], [[https://cython.org/][Cython]], [[https://pendulum.eustace.io/][Pendulum]] and [[https://github.com/Selfnet/hkp4py][hkp4py]]. 283 284Chances are quite high that at least the first one and maybe two of 285those will already be installed. 286 287Note that, as with Cython, some of advanced use case scenarios will 288bring with them additional requirements. Most of these will be fairly 289well known and commonly installed ones, however, which are in many 290cases likely to have already been installed on many systems or be 291familiar to Python programmers. 292 293 294** Installation 295 :PROPERTIES: 296 :CUSTOM_ID: installation 297 :END: 298 299Installing the Python bindings is effectively achieved by compiling 300and installing GPGME itself. 301 302Once SWIG is installed with Python and all the dependencies for GPGME 303are installed you only need to confirm that the version(s) of Python 304you want the bindings installed for are in your =$PATH=. 305 306By default GPGME will attempt to install the bindings for the most 307recent or highest version number of Python 2 and Python 3 it detects 308in =$PATH=. It specifically checks for the =python= and =python3= 309executables first and then checks for specific version numbers. 310 311For Python 2 it checks for these executables in this order: =python=, 312=python2= and =python2.7=. 313 314For Python 3 it checks for these executables in this order: =python3=, 315 =python3.7=, =python3.6=, =python3.5= and =python3.4=.[fn:2] 316 317On systems where =python= is actually =python3= and not =python2= it 318may be possible that =python2= may be overlooked, but there have been 319no reports of that actually occurring as yet. 320 321In the three months or so since the release of Python 3.7.0 there has 322been extensive testing and work with these bindings with no issues 323specifically relating to the new version of Python or any of the new 324features of either the language or the bindings. This has also been 325the case with Python 3.7.1rc1. With that in mind and given the 326release of Python 3.7.1 is scheduled for around the same time as GPGME 3271.12.0, the order of preferred Python versions has been changed to 328move Python 3.7 ahead of Python 3.6. 329 330 331*** Installing GPGME 332 :PROPERTIES: 333 :CUSTOM_ID: install-gpgme 334 :END: 335 336See the GPGME =README= file for details of how to install GPGME from 337source. 338 339 340** Known Issues 341 :PROPERTIES: 342 :CUSTOM_ID: snafu 343 :END: 344 345There are a few known issues with the current build process and the 346Python bindings. For the most part these are easily addressed should 347they be encountered. 348 349 350*** Breaking Builds 351 :PROPERTIES: 352 :CUSTOM_ID: snafu-a-swig-of-this-builds-character 353 :END: 354 355Occasionally when installing GPGME with the Python bindings included 356it may be observed that the =make= portion of that process induces a 357large very number of warnings and, eventually errors which end that 358part of the build process. Yet following that with =make check= and 359=make install= appears to work seamlessly. 360 361The cause of this is related to the way SWIG needs to be called to 362dynamically generate the C bindings for GPGME in the first place. So 363the entire process will always produce =lang/python/python2-gpg/= and 364=lang/python/python3-gpg/= directories. These should contain the 365build output generated during compilation, including the complete 366bindings and module installed into =site-packages=. 367 368Occasionally the errors in the early part or some other conflict 369(e.g. not installing as */root/* or */su/*) may result in nothing 370being installed to the relevant =site-packages= directory and the 371build directory missing a lot of expected files. Even when this 372occurs, the solution is actually quite simple and will always work. 373 374That solution is simply to run the following commands as either the 375*root* user or prepended with =sudo -H=[fn:3] in the =lang/python/= 376directory: 377 378#+BEGIN_SRC shell 379 /path/to/pythonX.Y setup.py build 380 /path/to/pythonX.Y setup.py build 381 /path/to/pythonX.Y setup.py install 382#+END_SRC 383 384Yes, the build command does need to be run twice. Yes, you still need 385to run the potentially failing or incomplete steps during the 386=configure=, =make= and =make install= steps with installing GPGME. 387This is because those steps generate a lot of essential files needed, 388both by and in order to create, the bindings (including both the 389=setup.py= and =gpgme.h= files). 390 391 392**** IMPORTANT Note 393 :PROPERTIES: 394 :CUSTOM_ID: snafu-swig-build-note 395 :END: 396 397If specifying a selected number of languages to create bindings for, 398try to leave Python last. Currently the majority of the other 399language bindings are also preceding Python of either version when 400listed alphabetically (not counting the Qt bindings). 401 402If Python is set to precede one of the other languages then it is 403possible that the errors described here may interrupt the build 404process before generating bindings for those other languages. In 405these cases it may be preferable to configure all preferred language 406bindings separately with alternative =configure= steps for GPGME using 407the =--enable-languages=$LANGUAGE= option. 408 409Alternatively =make= (or =gmake=, depending on your platform) may be 410run with the the =-k= option, which tells make to keep going even if 411errors are encountered. In that case the failure of one language's 412set of bindings to build should not hamper another language's bindings 413to build. 414 415 416*** Reinstalling Responsibly 417 :PROPERTIES: 418 :CUSTOM_ID: snafu-lessons-for-the-lazy 419 :END: 420 421Regardless of whether you're installing for one version of Python or 422several, there will come a point where reinstallation is required. 423With most Python module installations, the installed files go into the 424relevant site-packages directory and are then forgotten about. Then 425the module is upgraded, the new files are copied over the old and 426that's the end of the matter. 427 428While the same is true of these bindings, there have been intermittent 429issues observed on some platforms which have benefited significantly 430from removing all the previous installations of the bindings before 431installing the updated versions. 432 433Removing the previous version(s) is simply a matter of changing to the 434relevant =site-packages= directory for the version of Python in 435question and removing the =gpg/= directory and any accompanying 436egg-info files for that module. 437 438In most cases this will require root or administration privileges on 439the system, but the same is true of installing the module in the first 440place. 441 442 443*** Multiple installations 444 :PROPERTIES: 445 :CUSTOM_ID: snafu-the-full-monty 446 :END: 447 448For a variety of reasons it may be either necessary or just preferable 449to install the bindings to alternative installed Python versions which 450meet the requirements of these bindings. 451 452On POSIX systems this will generally be most simply achieved by 453running the manual installation commands (build, build, install) as 454described in the previous section for each Python installation the 455bindings need to be installed to. 456 457As per the SWIG documentation: the compilers, libraries and runtime 458used to build GPGME and the Python Bindings *must* match those used to 459compile Python itself, including the version number(s) (at least going 460by major version numbers and probably minor numbers too). 461 462On most POSIX systems, including OS X, this will very likely be the 463case in most, if not all, cases. 464 465Note that from GPGME [[https://dev.gnupg.org/rMff6ff616aea6f59b7f2ce1176492850ecdf3851e][1.12.1]] the default installation installs to each 466version of Python it can find first. That is that it will currently 467install for the first copies of Python versions 2.7, 3.4, 3.5, and so on 468up until the current dev branch that it finds. Usually this will be in the 469same prefix as GPGME itself, but is dictated by the =$PATH= when the 470installation is performed. The above instructions can still be 471performed on other python installations which the installer does not 472find, including alternative prefixes. 473 474 475 476*** Won't Work With Windows 477 :PROPERTIES: 478 :CUSTOM_ID: snafu-runtime-not-funtime 479 :END: 480 481There are semi-regular reports of Windows users having considerable 482difficulty in installing and using the Python bindings at all. Very 483often, possibly even always, these reports come from Cygwin users 484and/or MinGW users and/or Msys2 users. Though not all of them have 485been confirmed, it appears that these reports have also come from 486people who installed Python using the Windows installer files from the 487[[https://python.org][Python website]] (i.e. mostly MSI installers, sometimes self-extracting 488=.exe= files). 489 490The Windows versions of Python are not built using Cygwin, MinGW or 491Msys2; they're built using Microsoft Visual Studio. Furthermore the 492version used is /considerably/ more advanced than the version which 493MinGW obtained a small number of files from many years ago in order to 494be able to compile anything at all. Not only that, but there are 495changes to the version of Visual Studio between some micro releases, 496though that is is particularly the case with Python 2.7, since it has 497been kept around far longer than it should have been. 498 499There are two theoretical solutions to this issue: 500 501 1. Compile and install the GnuPG stack, including GPGME and the 502 Python bindings using the same version of Microsoft Visual Studio 503 used by the Python Foundation to compile the version of Python 504 installed. 505 506 If there are multiple versions of Python then this will need to be 507 done with each different version of Visual Studio used for those 508 versions of Python. 509 510 2. Compile and install Python using the same tools used by choice, 511 such as MinGW or Msys2. 512 513Do *not* use the official Windows installer for Python unless 514following the first method. 515 516In this type of situation it may even be for the best to accept that 517there are less limitations on permissive software than free software 518and simply opt to use a recent version of the Community Edition of 519Microsoft Visual Studio to compile and build all of it, no matter 520what. 521 522Investigations into the extent or the limitations of this issue are 523ongoing. 524 525The following table lists the version of Microsoft Visual Studio which 526needs to be used when compiling GPGME and the Python bindings with 527each version of the CPython binary released [[https://www.python.org/downloads/windows/][for Windows]]: 528 529| CPython | Microsoft product name | runtime filename | 530| 2.7.6 | Visual Studio 2008 | MSVCR90.DLL | 531| 3.4.0 | Visual Studio 2010 | MSVCR100.DLL | 532| 3.5.0 | Visual Studio 2015 | *see below* | 533| 3.6.0 | Visual Studio 2015 | *see below* | 534| 3.7.0 | Visual Studio 2017* | *see below* | 535 536It is important to note that MingW and Msys2 ship with the Visual C 537runtime from Microsoft Visual Studio 2005 and are thus *incompatible* 538with all the versions of CPython which can be used with the GPGME 539Python bindings. 540 541It is also important to note that from CPython 3.5 onwards, the Python 542Foundation has adopted the reworking of the Visual C runtime which was 543performed for Visual Studio 2015 and aimed at resolving many of these 544kinds of issues. Much greater detail on these issues and the correct 545file(s) to link to are available from Matthew Brett's invaluable page, 546[[https://matthew-brett.github.io/pydagogue/python_msvc.html][Using Microsoft Visual C with Python]]. It is also worth reading the 547Microsoft Developer Network blog post on [[http://blogs.msdn.com/b/vcblog/archive/2015/03/03/introducing-the-universal-crt.aspx][the universal CRT]] and Steve 548Dower's blog posts on Python extensions ([[http://stevedower.id.au/blog/building-for-python-3-5][part 1]] and [[http://stevedower.id.au/blog/building-for-python-3-5-part-two][part 2]]). 549 550The second of those two posts by Steve Dower contains the details of 551specific configuration options required for compiling anything to be 552used with official CPython releases. In addition to those 553configuration and compiler settings to use, the versions of Visual 554Studio prior to Visual Studio 2015 did not support 64-bit systems by 555default. So compiling a 64-bit version of these bindings for a 64-bit 556version of CPython 2.7 or 3.4 requires additional work. 557 558In addition to the blog posts, the [[https://wiki.python.org/moin/WindowsCompilers][Windows compilers]] wiki page on the 559CPython wiki is another essential reference on the relevant versions 560of Visual Studio to use and the degree of compatibility with CPython 561releases. 562 563Eventually someone will ask why there isn't an installable binary for 564Windows, which the GPGME of the licenses do not preclude as long as 565the source code is available in conjunction with such a release. 566 567The sheer number of versions of Visual Studio in conjunction with 568differing configuration options depending on the target Windows 569version and whether the architecture is 64-bit or 32-bit makes it 570difficult to provide a correct binary installer for Windows users. At 571the bare minimum doing so would require the GnuPG project compile ten 572different versions of the bindings with each release; both 32-bit and 57364-bit versions for CPython 2.7 and 3.4, with 64-bit versions for both 574x86-64 (i.e. Intel and AMD) and ARM architectures for CPython 3.5, 5753.6, 3.7 and later releases. That's the bare *minimum*, it'd probably 576be higher. 577 578Additionally, with only a binary installation used in conjunction with 579the CPython installer from =python.org= the advanced options available 580which utilise [[#cython][Cython]] will not be able to be used at all. Cython 581depends on being able to compile the C code it generates and that too 582would need to utilise a matching runtime to both the installed version 583of CPython and these bindings in order to work with the bindings. 584 585Considering all of that, what do we recommend? 586 587 1. Use a recent version of CPython; at least 3.5, but ideally 3.6 or 588 later. 589 590 2. Use Visual Studio 2015 or the standalone build tools for Visual 591 Studio 2017 (or later). 592 593 3. Compile both CPython and GPGME with these bindings using the tools 594 selected in step 2. 595 596 4. Ignore MingW, Msys2 and the official CPython binary installers. 597 598 5. Be thankful the answer to this question wasn't simply to say 599 something like, “install Linux” or “install FreeBSD” (or even 600 Apple's OS X). 601 602 603*** CFFI is the Best™ and GPGME should use it instead of SWIG 604 :PROPERTIES: 605 :CUSTOM_ID: snafu-cffi 606 :END: 607 608There are many reasons for favouring [[https://cffi.readthedocs.io/en/latest/overview.html][CFFI]] and proponents of it are 609quite happy to repeat these things as if all it would take to switch 610from SWIG to CFFI is repeating that list as if it were a new concept. 611 612The fact is that there are things which Python's CFFI implementation 613cannot handle in the GPGME C code. Beyond that there are features of 614SWIG which are simply not available with CFFI at all. SWIG generates 615the bindings to Python using the =gpgme.h= file, but that file is not 616a single version shipped with each release, it too is generated when 617GPGME is compiled. 618 619CFFI is currently unable to adapt to such a potentially mutable 620codebase. If there were some means of applying SWIG's dynamic code 621generation to produce the Python/CFFI API modes of accessing the GPGME 622libraries (or the source source code directly), but such a thing does 623not exist yet either and it currently appears that work is needed in 624at least one of CFFI's dependencies before any of this can be 625addressed. 626 627So if you're a massive fan of CFFI; that's great, but if you want this 628project to switch to CFFI then rather than just insisting that it 629should, I'd suggest you volunteer to bring CFFI up to the level this 630project needs. 631 632If you're actually seriously considering doing so, then I'd suggest 633taking the =gpgme-tool.c= file in the GPGME =src/= directory and 634getting that to work with any of the CFFI API methods (not the ABI 635methods, they'll work with pretty much anything). When you start 636running into trouble with "ifdefs" then you'll know what sort of 637things are lacking. That doesn't even take into account the amount of 638work saved via SWIG's code generation techniques either. 639 640 641*** Virtualised Environments 642 :PROPERTIES: 643 :CUSTOM_ID: snafu-venv 644 :END: 645 646It is fairly common practice amongst Python developers to, as much as 647possible, use packages like virtualenv to keep various things that are 648to be installed from interfering with each other. Given how much of 649the GPGME bindings is often at odds with the usual pythonic way of 650doing things, it stands to reason that this would be called into 651question too. 652 653As it happens the answer as to whether or not the bindings can be used 654with virtualenv, the answer is both yes and no. 655 656In general we recommend installing to the relevant path and matching 657prefix of GPGME itself. Which means that when GPGME, and ideally the 658rest of the GnuPG stack, is installed to a prefix like =/usr/local= or 659=/opt/local= then the bindings would need to be installed to the main 660Python installation and not a virtualised abstraction. Attempts to 661separate the two in the past have been known to cause weird and 662intermittent errors ranging from minor annoyances to complete failures 663in the build process. 664 665As a consequence we only recommend building with and installing to the 666main Python installations within the same prefix as GPGME is installed 667to or which are found by GPGME's configuration stage immediately prior 668to running the make commands. Which is exactly what the compiling and 669installing process of GPGME does by default. 670 671Once that is done, however, it appears that a copy of the compiled 672module may be installed into a virtualenv of the same major and minor 673version matching the build. Alternatively it is possible to utilise a 674=sites.pth= file in the =site-packages/= directory of a virtualenv 675installation, which links back to the system installations 676corresponding directory in order to import anything installed system 677wide. This may or may not be appropriate on a case by case basis. 678 679Though extensive testing of either of these options is not yet 680complete, preliminary testing of them indicates that both are viable 681as long as the main installation is complete. Which means that 682certain other options normally restricted to virtual environments are 683also available, including integration with pythonic test suites 684(e.g. [[https://docs.pytest.org/en/latest/index.html][pytest]]) and other large projects. 685 686That said, it is worth reiterating the warning regarding non-standard 687installations. If one were to attempt to install the bindings only to 688a virtual environment without somehow also including the full GnuPG 689stack (or enough of it as to include GPGME) then it is highly likely 690that errors would be encountered at some point and more than a little 691likely that the build process itself would break. 692 693If a degree of separation from the main operating system is still 694required in spite of these warnings, then consider other forms of 695virtualisation. Either a virtual machine (e.g. [[https://www.virtualbox.org/][VirtualBox]]), a 696hardware emulation layer (e.g. [[https://www.qemu.org/][QEMU]]) or an application container 697(e.g. [[https://www.docker.com/why-docker][Docker]]). 698 699Finally it should be noted that the limited tests conducted thus far 700have been using the =virtualenv= command in a new directory to create 701the virtual python environment. As opposed to the standard =python3 702-m venv= and it is possible that this will make a difference depending 703on the system and version of Python in use. Another option is to run 704the command =python3 -m virtualenv /path/to/install/virtual/thingy= 705instead. 706 707 708*** Post installation 709 :PROPERTIES: 710 :CUSTOM_ID: snafu-docs 711 :END: 712 713Following installation it is recommended to move the 714=post_installer.py= script from the =lang/python/examples/howto/= 715directory to the =lang/python/= directory and run it. This will fix 716or restore files needed by Sphinx which may be removed during a 717distribution build for release. It will also generate reST files from 718Org mode files with Pandoc and generate Texinfo files from Org mode 719files with GNU Emacs and Org mode (in batch mode). Additionally it 720will fix the UTF-8 declaration line in the Texinfo files (Emacs 721expects "UTF-8" to be "utf-8"). 722 723 724* Fundamentals 725 :PROPERTIES: 726 :CUSTOM_ID: howto-fund-a-mental 727 :END: 728 729Before we can get to the fun stuff, there are a few matters regarding 730GPGME's design which hold true whether you're dealing with the C code 731directly or these Python bindings. 732 733 734** No REST 735 :PROPERTIES: 736 :CUSTOM_ID: no-rest-for-the-wicked 737 :END: 738 739The first part of which is or will be fairly blatantly obvious upon 740viewing the first example, but it's worth reiterating anyway. That 741being that this API is /*not*/ a REST API. Nor indeed could it ever 742be one. 743 744Most, if not all, Python programmers (and not just Python programmers) 745know how easy it is to work with a RESTful API. In fact they've 746become so popular that many other APIs attempt to emulate REST-like 747behaviour as much as they are able. Right down to the use of JSON 748formatted output to facilitate the use of their API without having to 749retrain developers. 750 751This API does not do that. It would not be able to do that and also 752provide access to the entire C API on which it's built. It does, 753however, provide a very pythonic interface on top of the direct 754bindings and it's this pythonic layer that this HOWTO deals with. 755 756 757** Context 758 :PROPERTIES: 759 :CUSTOM_ID: howto-get-context 760 :END: 761 762One of the reasons which prevents this API from being RESTful is that 763most operations require more than one instruction to the API to 764perform the task. Sure, there are certain functions which can be 765performed simultaneously, particularly if the result known or strongly 766anticipated (e.g. selecting and encrypting to a key known to be in the 767public keybox). 768 769There are many more, however, which cannot be manipulated so readily: 770they must be performed in a specific sequence and the result of one 771operation has a direct bearing on the outcome of subsequent 772operations. Not merely by generating an error either. 773 774When dealing with this type of persistent state on the web, full of 775both the RESTful and REST-like, it's most commonly referred to as a 776session. In GPGME, however, it is called a context and every 777operation type has one. 778 779 780* Working with keys 781 :PROPERTIES: 782 :CUSTOM_ID: howto-keys 783 :END: 784 785 786** Key selection 787 :PROPERTIES: 788 :CUSTOM_ID: howto-keys-selection 789 :END: 790 791Selecting keys to encrypt to or to sign with will be a common 792occurrence when working with GPGMe and the means available for doing 793so are quite simple. 794 795They do depend on utilising a Context; however once the data is 796recorded in another variable, that Context does not need to be the 797same one which subsequent operations are performed. 798 799The easiest way to select a specific key is by searching for that 800key's key ID or fingerprint, preferably the full fingerprint without 801any spaces in it. A long key ID will probably be okay, but is not 802advised and short key IDs are already a problem with some being 803generated to match specific patterns. It does not matter whether the 804pattern is upper or lower case. 805 806So this is the best method: 807 808#+BEGIN_SRC python -i 809import gpg 810 811k = gpg.Context().keylist(pattern="258E88DCBD3CD44D8E7AB43F6ECB6AF0DEADBEEF") 812keys = list(k) 813#+END_SRC 814 815This is passable and very likely to be common: 816 817#+BEGIN_SRC python -i 818import gpg 819 820k = gpg.Context().keylist(pattern="0x6ECB6AF0DEADBEEF") 821keys = list(k) 822#+END_SRC 823 824And this is a really bad idea: 825 826#+BEGIN_SRC python -i 827import gpg 828 829k = gpg.Context().keylist(pattern="0xDEADBEEF") 830keys = list(k) 831#+END_SRC 832 833Alternatively it may be that the intention is to create a list of keys 834which all match a particular search string. For instance all the 835addresses at a particular domain, like this: 836 837#+BEGIN_SRC python -i 838import gpg 839 840ncsc = gpg.Context().keylist(pattern="ncsc.mil") 841nsa = list(ncsc) 842#+END_SRC 843 844 845*** Counting keys 846 :PROPERTIES: 847 :CUSTOM_ID: howto-keys-counting 848 :END: 849 850Counting the number of keys in your public keybox (=pubring.kbx=), the 851format which has superseded the old keyring format (=pubring.gpg= and 852=secring.gpg=), or the number of secret keys is a very simple task. 853 854#+BEGIN_SRC python -i 855import gpg 856 857c = gpg.Context() 858seckeys = c.keylist(pattern=None, secret=True) 859pubkeys = c.keylist(pattern=None, secret=False) 860 861seclist = list(seckeys) 862secnum = len(seclist) 863 864publist = list(pubkeys) 865pubnum = len(publist) 866 867print(""" 868 Number of secret keys: {0} 869 Number of public keys: {1} 870""".format(secnum, pubnum)) 871#+END_SRC 872 873NOTE: The [[#cython][Cython]] introduction in the [[#advanced-use][Advanced and Experimental]] 874section uses this same key counting code with Cython to demonstrate 875some areas where Cython can improve performance even with the 876bindings. Users with large public keyrings or keyboxes, for instance, 877should consider these options if they are comfortable with using 878Cython. 879 880 881** Get key 882 :PROPERTIES: 883 :CUSTOM_ID: howto-get-key 884 :END: 885 886An alternative method of getting a single key via its fingerprint is 887available directly within a Context with =Context().get_key=. This is 888the preferred method of selecting a key in order to modify it, sign or 889certify it and for obtaining relevant data about a single key as a 890part of other functions; when verifying a signature made by that key, 891for instance. 892 893By default this method will select public keys, but it can select 894secret keys as well. 895 896This first example demonstrates selecting the current key of Werner 897Koch, which is due to expire at the end of 2018: 898 899#+BEGIN_SRC python -i 900import gpg 901 902fingerprint = "80615870F5BAD690333686D0F2AD85AC1E42B367" 903key = gpg.Context().get_key(fingerprint) 904#+END_SRC 905 906Whereas this example demonstrates selecting the author's current key 907with the =secret= key word argument set to =True=: 908 909#+BEGIN_SRC python -i 910import gpg 911 912fingerprint = "DB4724E6FA4286C92B4E55C4321E4E2373590E5D" 913key = gpg.Context().get_key(fingerprint, secret=True) 914#+END_SRC 915 916It is, of course, quite possible to select expired, disabled and 917revoked keys with this function, but only to effectively display 918information about those keys. 919 920It is also possible to use both unicode or string literals and byte 921literals with the fingerprint when getting a key in this way. 922 923 924** Importing keys 925 :PROPERTIES: 926 :CUSTOM_ID: howto-import-key 927 :END: 928 929Importing keys is possible with the =key_import()= method and takes 930one argument which is a bytes literal object containing either the 931binary or ASCII armoured key data for one or more keys. 932 933The following example retrieves one or more keys from the SKS 934keyservers via the web using the requests module. Since requests 935returns the content as a bytes literal object, we can then use that 936directly to import the resulting data into our keybox. 937 938#+BEGIN_SRC python -i 939import gpg 940import os.path 941import requests 942 943c = gpg.Context() 944url = "https://sks-keyservers.net/pks/lookup" 945pattern = input("Enter the pattern to search for key or user IDs: ") 946payload = {"op": "get", "search": pattern} 947 948r = requests.get(url, verify=True, params=payload) 949result = c.key_import(r.content) 950 951if result is not None and hasattr(result, "considered") is False: 952 print(result) 953elif result is not None and hasattr(result, "considered") is True: 954 num_keys = len(result.imports) 955 new_revs = result.new_revocations 956 new_sigs = result.new_signatures 957 new_subs = result.new_sub_keys 958 new_uids = result.new_user_ids 959 new_scrt = result.secret_imported 960 nochange = result.unchanged 961 print(""" 962 The total number of keys considered for import was: {0} 963 964 Number of keys revoked: {1} 965 Number of new signatures: {2} 966 Number of new subkeys: {3} 967 Number of new user IDs: {4} 968 Number of new secret keys: {5} 969 Number of unchanged keys: {6} 970 971 The key IDs for all considered keys were: 972""".format(num_keys, new_revs, new_sigs, new_subs, new_uids, new_scrt, 973 nochange)) 974 for i in range(num_keys): 975 print("{0}\n".format(result.imports[i].fpr)) 976else: 977 pass 978#+END_SRC 979 980NOTE: When searching for a key ID of any length or a fingerprint 981(without spaces), the SKS servers require the the leading =0x= 982indicative of hexadecimal be included. Also note that the old short 983key IDs (e.g. =0xDEADBEEF=) should no longer be used due to the 984relative ease by which such key IDs can be reproduced, as demonstrated 985by the Evil32 Project in 2014 (which was subsequently exploited in 9862016). 987 988Testing for whether a string in any given search is or may be a 989hexadecimal value which may be missing the leading =0x= is a simple 990matter of using a try/except statement which attempts to convert the 991string as hex to an integer and then back to hex; then using that to 992search with. Raising a ValueError simply results in treating the 993string as a string. This is the method and logic utilised in the 994=import-keys-hkp.py= script (see below). 995 996 997*** Working with ProtonMail 998 :PROPERTIES: 999 :CUSTOM_ID: import-protonmail 1000 :END: 1001 1002Here is a variation on the example above which checks the constrained 1003ProtonMail keyserver for ProtonMail public keys. 1004 1005#+BEGIN_SRC python -i 1006import gpg 1007import requests 1008import sys 1009 1010print(""" 1011This script searches the ProtonMail key server for the specified key and 1012imports it. 1013""") 1014 1015c = gpg.Context(armor=True) 1016url = "https://api.protonmail.ch/pks/lookup" 1017ksearch = [] 1018 1019if len(sys.argv) >= 2: 1020 keyterm = sys.argv[1] 1021else: 1022 keyterm = input("Enter the key ID, UID or search string: ") 1023 1024if keyterm.count("@") == 2 and keyterm.startswith("@") is True: 1025 ksearch.append(keyterm[1:]) 1026 ksearch.append(keyterm[1:]) 1027 ksearch.append(keyterm[1:]) 1028elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: 1029 ksearch.append("{0}@protonmail.com".format(keyterm[1:])) 1030 ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) 1031 ksearch.append("{0}@pm.me".format(keyterm[1:])) 1032elif keyterm.count("@") == 0: 1033 ksearch.append("{0}@protonmail.com".format(keyterm)) 1034 ksearch.append("{0}@protonmail.ch".format(keyterm)) 1035 ksearch.append("{0}@pm.me".format(keyterm)) 1036elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: 1037 uidlist = keyterm.split("@") 1038 for uid in uidlist: 1039 ksearch.append("{0}@protonmail.com".format(uid)) 1040 ksearch.append("{0}@protonmail.ch".format(uid)) 1041 ksearch.append("{0}@pm.me".format(uid)) 1042elif keyterm.count("@") > 2: 1043 uidlist = keyterm.split("@") 1044 for uid in uidlist: 1045 ksearch.append("{0}@protonmail.com".format(uid)) 1046 ksearch.append("{0}@protonmail.ch".format(uid)) 1047 ksearch.append("{0}@pm.me".format(uid)) 1048else: 1049 ksearch.append(keyterm) 1050 1051for k in ksearch: 1052 payload = {"op": "get", "search": k} 1053 try: 1054 r = requests.get(url, verify=True, params=payload) 1055 if r.ok is True: 1056 result = c.key_import(r.content) 1057 elif r.ok is False: 1058 result = r.content 1059 except Exception as e: 1060 result = None 1061 1062 if result is not None and hasattr(result, "considered") is False: 1063 print("{0} for {1}".format(result.decode(), k)) 1064 elif result is not None and hasattr(result, "considered") is True: 1065 num_keys = len(result.imports) 1066 new_revs = result.new_revocations 1067 new_sigs = result.new_signatures 1068 new_subs = result.new_sub_keys 1069 new_uids = result.new_user_ids 1070 new_scrt = result.secret_imported 1071 nochange = result.unchanged 1072 print(""" 1073The total number of keys considered for import was: {0} 1074 1075With UIDs wholely or partially matching the following string: 1076 1077 {1} 1078 1079 Number of keys revoked: {2} 1080 Number of new signatures: {3} 1081 Number of new subkeys: {4} 1082 Number of new user IDs: {5} 1083Number of new secret keys: {6} 1084 Number of unchanged keys: {7} 1085 1086The key IDs for all considered keys were: 1087""".format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, 1088 nochange)) 1089 for i in range(num_keys): 1090 print(result.imports[i].fpr) 1091 print("") 1092 elif result is None: 1093 print(e) 1094#+END_SRC 1095 1096Both the above example, [[../examples/howto/pmkey-import.py][pmkey-import.py]], and a version which prompts 1097for an alternative GnuPG home directory, [[../examples/howto/pmkey-import-alt.py][pmkey-import-alt.py]], are 1098available with the other examples and are executable scripts. 1099 1100Note that while the ProtonMail servers are based on the SKS servers, 1101their server is related more to their API and is not feature complete 1102by comparison to the servers in the SKS pool. One notable difference 1103being that the ProtonMail server does not permit non ProtonMail users 1104to update their own keys, which could be a vector for attacking 1105ProtonMail users who may not receive a key's revocation if it had been 1106compromised. 1107 1108 1109*** Importing with HKP for Python 1110 :PROPERTIES: 1111 :CUSTOM_ID: import-hkp4py 1112 :END: 1113 1114Performing the same tasks with the [[https://github.com/Selfnet/hkp4py][hkp4py module]] (available via PyPI) 1115is not too much different, but does provide a number of options of 1116benefit to end users. Not least of which being the ability to perform 1117some checks on a key before importing it or not. For instance it may 1118be the policy of a site or project to only import keys which have not 1119been revoked. The hkp4py module permits such checks prior to the 1120importing of the keys found. 1121 1122#+BEGIN_SRC python -i 1123import gpg 1124import hkp4py 1125import sys 1126 1127c = gpg.Context() 1128server = hkp4py.KeyServer("hkps://hkps.pool.sks-keyservers.net") 1129results = [] 1130keys = [] 1131 1132if len(sys.argv) > 2: 1133 pattern = " ".join(sys.argv[1:]) 1134elif len(sys.argv) == 2: 1135 pattern = sys.argv[1] 1136else: 1137 pattern = input("Enter the pattern to search for keys or user IDs: ") 1138 1139 1140if pattern is not None: 1141 try: 1142 key = server.search(hex(int(pattern, 16))) 1143 keyed = True 1144 except ValueError as ve: 1145 key = server.search(pattern) 1146 keyed = False 1147 1148 if key is not None: 1149 keys.append(key[0]) 1150 if keyed is True: 1151 try: 1152 fob = server.search(pattern) 1153 except: 1154 fob = None 1155 if fob is not None: 1156 keys.append(fob[0]) 1157 else: 1158 pass 1159 else: 1160 pass 1161 1162 for logrus in pattern.split(): 1163 try: 1164 key = server.search(hex(int(logrus, 16))) 1165 hexed = True 1166 except ValueError as ve: 1167 key = server.search(logrus) 1168 hexed = False 1169 1170 if key is not None: 1171 keys.append(key[0]) 1172 if hexed is True: 1173 try: 1174 fob = server.search(logrus) 1175 except: 1176 fob = None 1177 if fob is not None: 1178 keys.append(fob[0]) 1179 else: 1180 pass 1181 else: 1182 pass 1183 1184 1185if len(keys) > 0: 1186 for key in keys: 1187 import_result = c.key_import(key.key_blob) 1188 results.append(import_result) 1189 1190for result in results: 1191 if result is not None and hasattr(result, "considered") is False: 1192 print(result) 1193 elif result is not None and hasattr(result, "considered") is True: 1194 num_keys = len(result.imports) 1195 new_revs = result.new_revocations 1196 new_sigs = result.new_signatures 1197 new_subs = result.new_sub_keys 1198 new_uids = result.new_user_ids 1199 new_scrt = result.secret_imported 1200 nochange = result.unchanged 1201 print(""" 1202The total number of keys considered for import was: {0} 1203 1204 Number of keys revoked: {1} 1205 Number of new signatures: {2} 1206 Number of new subkeys: {3} 1207 Number of new user IDs: {4} 1208Number of new secret keys: {5} 1209 Number of unchanged keys: {6} 1210 1211The key IDs for all considered keys were: 1212""".format(num_keys, new_revs, new_sigs, new_subs, new_uids, new_scrt, 1213 nochange)) 1214 for i in range(num_keys): 1215 print(result.imports[i].fpr) 1216 print("") 1217 else: 1218 pass 1219#+END_SRC 1220 1221Since the hkp4py module handles multiple keys just as effectively as 1222one (=keys= is a list of responses per matching key), the example 1223above is able to do a little bit more with the returned data before 1224anything is actually imported. 1225 1226 1227*** Importing from ProtonMail with HKP for Python 1228 :PROPERTIES: 1229 :CUSTOM_ID: import-protonmail-hkp4py 1230 :END: 1231 1232Though this can provide certain benefits even when working with 1233ProtonMail, the scope is somewhat constrained there due to the 1234limitations of the ProtonMail keyserver. 1235 1236For instance, searching the SKS keyserver pool for the term "gnupg" 1237produces hundreds of results from any time the word appears in any 1238part of a user ID. Performing the same search on the ProtonMail 1239keyserver returns zero results, even though there are at least two 1240test accounts which include it as part of the username. 1241 1242The cause of this discrepancy is the deliberate configuration of that 1243server by ProtonMail to require an exact match of the full email 1244address of the ProtonMail user whose key is being requested. 1245Presumably this is intended to reduce breaches of privacy of their 1246users as an email address must already be known before a key for that 1247address can be obtained. 1248 1249 1250**** Import from ProtonMail via HKP for Python Example no. 1 1251 :PROPERTIES: 1252 :CUSTOM_ID: import-hkp4py-pm1 1253 :END: 1254 1255The following script is available with the rest of the examples under 1256the somewhat less than original name, =pmkey-import-hkp.py=. 1257 1258#+BEGIN_SRC python -i 1259import gpg 1260import hkp4py 1261import os.path 1262import sys 1263 1264print(""" 1265This script searches the ProtonMail key server for the specified key and 1266imports it. 1267 1268Usage: pmkey-import-hkp.py [search strings] 1269""") 1270 1271c = gpg.Context(armor=True) 1272server = hkp4py.KeyServer("hkps://api.protonmail.ch") 1273keyterms = [] 1274ksearch = [] 1275allkeys = [] 1276results = [] 1277paradox = [] 1278homeless = None 1279 1280if len(sys.argv) > 2: 1281 keyterms = sys.argv[1:] 1282elif len(sys.argv) == 2: 1283 keyterm = sys.argv[1] 1284 keyterms.append(keyterm) 1285else: 1286 key_term = input("Enter the key ID, UID or search string: ") 1287 keyterms = key_term.split() 1288 1289for keyterm in keyterms: 1290 if keyterm.count("@") == 2 and keyterm.startswith("@") is True: 1291 ksearch.append(keyterm[1:]) 1292 ksearch.append(keyterm[1:]) 1293 ksearch.append(keyterm[1:]) 1294 elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: 1295 ksearch.append("{0}@protonmail.com".format(keyterm[1:])) 1296 ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) 1297 ksearch.append("{0}@pm.me".format(keyterm[1:])) 1298 elif keyterm.count("@") == 0: 1299 ksearch.append("{0}@protonmail.com".format(keyterm)) 1300 ksearch.append("{0}@protonmail.ch".format(keyterm)) 1301 ksearch.append("{0}@pm.me".format(keyterm)) 1302 elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: 1303 uidlist = keyterm.split("@") 1304 for uid in uidlist: 1305 ksearch.append("{0}@protonmail.com".format(uid)) 1306 ksearch.append("{0}@protonmail.ch".format(uid)) 1307 ksearch.append("{0}@pm.me".format(uid)) 1308 elif keyterm.count("@") > 2: 1309 uidlist = keyterm.split("@") 1310 for uid in uidlist: 1311 ksearch.append("{0}@protonmail.com".format(uid)) 1312 ksearch.append("{0}@protonmail.ch".format(uid)) 1313 ksearch.append("{0}@pm.me".format(uid)) 1314 else: 1315 ksearch.append(keyterm) 1316 1317for k in ksearch: 1318 print("Checking for key for: {0}".format(k)) 1319 try: 1320 keys = server.search(k) 1321 if isinstance(keys, list) is True: 1322 for key in keys: 1323 allkeys.append(key) 1324 try: 1325 import_result = c.key_import(key.key_blob) 1326 except Exception as e: 1327 import_result = c.key_import(key.key) 1328 else: 1329 paradox.append(keys) 1330 import_result = None 1331 except Exception as e: 1332 import_result = None 1333 results.append(import_result) 1334 1335for result in results: 1336 if result is not None and hasattr(result, "considered") is False: 1337 print("{0} for {1}".format(result.decode(), k)) 1338 elif result is not None and hasattr(result, "considered") is True: 1339 num_keys = len(result.imports) 1340 new_revs = result.new_revocations 1341 new_sigs = result.new_signatures 1342 new_subs = result.new_sub_keys 1343 new_uids = result.new_user_ids 1344 new_scrt = result.secret_imported 1345 nochange = result.unchanged 1346 print(""" 1347The total number of keys considered for import was: {0} 1348 1349With UIDs wholely or partially matching the following string: 1350 1351 {1} 1352 1353 Number of keys revoked: {2} 1354 Number of new signatures: {3} 1355 Number of new subkeys: {4} 1356 Number of new user IDs: {5} 1357Number of new secret keys: {6} 1358 Number of unchanged keys: {7} 1359 1360The key IDs for all considered keys were: 1361""".format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, 1362 nochange)) 1363 for i in range(num_keys): 1364 print(result.imports[i].fpr) 1365 print("") 1366 elif result is None: 1367 pass 1368#+END_SRC 1369 1370 1371**** Import from ProtonMail via HKP for Python Example no. 2 1372 :PROPERTIES: 1373 :CUSTOM_ID: import-hkp4py-pm2 1374 :END: 1375 1376Like its counterpart above, this script can also be found with the 1377rest of the examples, by the name pmkey-import-hkp-alt.py. 1378 1379With this script a modicum of effort has been made to treat anything 1380passed as a =homedir= which either does not exist or which is not a 1381directory, as also being a pssible user ID to check for. It's not 1382guaranteed to pick up on all such cases, but it should cover most of 1383them. 1384 1385#+BEGIN_SRC python -i 1386import gpg 1387import hkp4py 1388import os.path 1389import sys 1390 1391print(""" 1392This script searches the ProtonMail key server for the specified key and 1393imports it. Optionally enables specifying a different GnuPG home directory. 1394 1395Usage: pmkey-import-hkp.py [homedir] [search string] 1396 or: pmkey-import-hkp.py [search string] 1397""") 1398 1399c = gpg.Context(armor=True) 1400server = hkp4py.KeyServer("hkps://api.protonmail.ch") 1401keyterms = [] 1402ksearch = [] 1403allkeys = [] 1404results = [] 1405paradox = [] 1406homeless = None 1407 1408if len(sys.argv) > 3: 1409 homedir = sys.argv[1] 1410 keyterms = sys.argv[2:] 1411elif len(sys.argv) == 3: 1412 homedir = sys.argv[1] 1413 keyterm = sys.argv[2] 1414 keyterms.append(keyterm) 1415elif len(sys.argv) == 2: 1416 homedir = "" 1417 keyterm = sys.argv[1] 1418 keyterms.append(keyterm) 1419else: 1420 keyterm = input("Enter the key ID, UID or search string: ") 1421 homedir = input("Enter the GPG configuration directory path (optional): ") 1422 keyterms.append(keyterm) 1423 1424if len(homedir) == 0: 1425 homedir = None 1426 homeless = False 1427 1428if homedir is not None: 1429 if homedir.startswith("~"): 1430 if os.path.exists(os.path.expanduser(homedir)) is True: 1431 if os.path.isdir(os.path.expanduser(homedir)) is True: 1432 c.home_dir = os.path.realpath(os.path.expanduser(homedir)) 1433 else: 1434 homeless = True 1435 else: 1436 homeless = True 1437 elif os.path.exists(os.path.realpath(homedir)) is True: 1438 if os.path.isdir(os.path.realpath(homedir)) is True: 1439 c.home_dir = os.path.realpath(homedir) 1440 else: 1441 homeless = True 1442 else: 1443 homeless = True 1444 1445# First check to see if the homedir really is a homedir and if not, treat it as 1446# a search string. 1447if homeless is True: 1448 keyterms.append(homedir) 1449 c.home_dir = None 1450else: 1451 pass 1452 1453for keyterm in keyterms: 1454 if keyterm.count("@") == 2 and keyterm.startswith("@") is True: 1455 ksearch.append(keyterm[1:]) 1456 ksearch.append(keyterm[1:]) 1457 ksearch.append(keyterm[1:]) 1458 elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: 1459 ksearch.append("{0}@protonmail.com".format(keyterm[1:])) 1460 ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) 1461 ksearch.append("{0}@pm.me".format(keyterm[1:])) 1462 elif keyterm.count("@") == 0: 1463 ksearch.append("{0}@protonmail.com".format(keyterm)) 1464 ksearch.append("{0}@protonmail.ch".format(keyterm)) 1465 ksearch.append("{0}@pm.me".format(keyterm)) 1466 elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: 1467 uidlist = keyterm.split("@") 1468 for uid in uidlist: 1469 ksearch.append("{0}@protonmail.com".format(uid)) 1470 ksearch.append("{0}@protonmail.ch".format(uid)) 1471 ksearch.append("{0}@pm.me".format(uid)) 1472 elif keyterm.count("@") > 2: 1473 uidlist = keyterm.split("@") 1474 for uid in uidlist: 1475 ksearch.append("{0}@protonmail.com".format(uid)) 1476 ksearch.append("{0}@protonmail.ch".format(uid)) 1477 ksearch.append("{0}@pm.me".format(uid)) 1478 else: 1479 ksearch.append(keyterm) 1480 1481for k in ksearch: 1482 print("Checking for key for: {0}".format(k)) 1483 try: 1484 keys = server.search(k) 1485 if isinstance(keys, list) is True: 1486 for key in keys: 1487 allkeys.append(key) 1488 try: 1489 import_result = c.key_import(key.key_blob) 1490 except Exception as e: 1491 import_result = c.key_import(key.key) 1492 else: 1493 paradox.append(keys) 1494 import_result = None 1495 except Exception as e: 1496 import_result = None 1497 results.append(import_result) 1498 1499for result in results: 1500 if result is not None and hasattr(result, "considered") is False: 1501 print("{0} for {1}".format(result.decode(), k)) 1502 elif result is not None and hasattr(result, "considered") is True: 1503 num_keys = len(result.imports) 1504 new_revs = result.new_revocations 1505 new_sigs = result.new_signatures 1506 new_subs = result.new_sub_keys 1507 new_uids = result.new_user_ids 1508 new_scrt = result.secret_imported 1509 nochange = result.unchanged 1510 print(""" 1511The total number of keys considered for import was: {0} 1512 1513With UIDs wholely or partially matching the following string: 1514 1515 {1} 1516 1517 Number of keys revoked: {2} 1518 Number of new signatures: {3} 1519 Number of new subkeys: {4} 1520 Number of new user IDs: {5} 1521Number of new secret keys: {6} 1522 Number of unchanged keys: {7} 1523 1524The key IDs for all considered keys were: 1525""".format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, 1526 nochange)) 1527 for i in range(num_keys): 1528 print(result.imports[i].fpr) 1529 print("") 1530 elif result is None: 1531 pass 1532#+END_SRC 1533 1534 1535** Exporting keys 1536 :PROPERTIES: 1537 :CUSTOM_ID: howto-export-key 1538 :END: 1539 1540Exporting keys remains a reasonably simple task, but has been 1541separated into three different functions for the OpenPGP cryptographic 1542engine. Two of those functions are for exporting public keys and the 1543third is for exporting secret keys. 1544 1545 1546*** Exporting public keys 1547 :PROPERTIES: 1548 :CUSTOM_ID: howto-export-public-key 1549 :END: 1550 1551There are two methods of exporting public keys, both of which are very 1552similar to the other. The default method, =key_export()=, will export 1553a public key or keys matching a specified pattern as normal. The 1554alternative, the =key_export_minimal()= method, will do the same thing 1555except producing a minimised output with extra signatures and third 1556party signatures or certifications removed. 1557 1558#+BEGIN_SRC python -i 1559import gpg 1560import os.path 1561import sys 1562 1563print(""" 1564This script exports one or more public keys. 1565""") 1566 1567c = gpg.Context(armor=True) 1568 1569if len(sys.argv) >= 4: 1570 keyfile = sys.argv[1] 1571 logrus = sys.argv[2] 1572 homedir = sys.argv[3] 1573elif len(sys.argv) == 3: 1574 keyfile = sys.argv[1] 1575 logrus = sys.argv[2] 1576 homedir = input("Enter the GPG configuration directory path (optional): ") 1577elif len(sys.argv) == 2: 1578 keyfile = sys.argv[1] 1579 logrus = input("Enter the UID matching the key(s) to export: ") 1580 homedir = input("Enter the GPG configuration directory path (optional): ") 1581else: 1582 keyfile = input("Enter the path and filename to save the secret key to: ") 1583 logrus = input("Enter the UID matching the key(s) to export: ") 1584 homedir = input("Enter the GPG configuration directory path (optional): ") 1585 1586if homedir.startswith("~"): 1587 if os.path.exists(os.path.expanduser(homedir)) is True: 1588 c.home_dir = os.path.expanduser(homedir) 1589 else: 1590 pass 1591elif os.path.exists(homedir) is True: 1592 c.home_dir = homedir 1593else: 1594 pass 1595 1596try: 1597 result = c.key_export(pattern=logrus) 1598except: 1599 result = c.key_export(pattern=None) 1600 1601if result is not None: 1602 with open(keyfile, "wb") as f: 1603 f.write(result) 1604else: 1605 pass 1606#+END_SRC 1607 1608It should be noted that the result will only return =None= when a 1609search pattern has been entered, but has not matched any keys. When 1610the search pattern itself is set to =None= this triggers the exporting 1611of the entire public keybox. 1612 1613#+BEGIN_SRC python -i 1614import gpg 1615import os.path 1616import sys 1617 1618print(""" 1619This script exports one or more public keys in minimised form. 1620""") 1621 1622c = gpg.Context(armor=True) 1623 1624if len(sys.argv) >= 4: 1625 keyfile = sys.argv[1] 1626 logrus = sys.argv[2] 1627 homedir = sys.argv[3] 1628elif len(sys.argv) == 3: 1629 keyfile = sys.argv[1] 1630 logrus = sys.argv[2] 1631 homedir = input("Enter the GPG configuration directory path (optional): ") 1632elif len(sys.argv) == 2: 1633 keyfile = sys.argv[1] 1634 logrus = input("Enter the UID matching the key(s) to export: ") 1635 homedir = input("Enter the GPG configuration directory path (optional): ") 1636else: 1637 keyfile = input("Enter the path and filename to save the secret key to: ") 1638 logrus = input("Enter the UID matching the key(s) to export: ") 1639 homedir = input("Enter the GPG configuration directory path (optional): ") 1640 1641if homedir.startswith("~"): 1642 if os.path.exists(os.path.expanduser(homedir)) is True: 1643 c.home_dir = os.path.expanduser(homedir) 1644 else: 1645 pass 1646elif os.path.exists(homedir) is True: 1647 c.home_dir = homedir 1648else: 1649 pass 1650 1651try: 1652 result = c.key_export_minimal(pattern=logrus) 1653except: 1654 result = c.key_export_minimal(pattern=None) 1655 1656if result is not None: 1657 with open(keyfile, "wb") as f: 1658 f.write(result) 1659else: 1660 pass 1661#+END_SRC 1662 1663 1664*** Exporting secret keys 1665 :PROPERTIES: 1666 :CUSTOM_ID: howto-export-secret-key 1667 :END: 1668 1669Exporting secret keys is, functionally, very similar to exporting 1670public keys; save for the invocation of =pinentry= via =gpg-agent= in 1671order to securely enter the key's passphrase and authorise the export. 1672 1673The following example exports the secret key to a file which is then 1674set with the same permissions as the output files created by the 1675command line secret key export options. 1676 1677#+BEGIN_SRC python -i 1678import gpg 1679import os 1680import os.path 1681import sys 1682 1683print(""" 1684This script exports one or more secret keys. 1685 1686The gpg-agent and pinentry are invoked to authorise the export. 1687""") 1688 1689c = gpg.Context(armor=True) 1690 1691if len(sys.argv) >= 4: 1692 keyfile = sys.argv[1] 1693 logrus = sys.argv[2] 1694 homedir = sys.argv[3] 1695elif len(sys.argv) == 3: 1696 keyfile = sys.argv[1] 1697 logrus = sys.argv[2] 1698 homedir = input("Enter the GPG configuration directory path (optional): ") 1699elif len(sys.argv) == 2: 1700 keyfile = sys.argv[1] 1701 logrus = input("Enter the UID matching the secret key(s) to export: ") 1702 homedir = input("Enter the GPG configuration directory path (optional): ") 1703else: 1704 keyfile = input("Enter the path and filename to save the secret key to: ") 1705 logrus = input("Enter the UID matching the secret key(s) to export: ") 1706 homedir = input("Enter the GPG configuration directory path (optional): ") 1707 1708if len(homedir) == 0: 1709 homedir = None 1710elif homedir.startswith("~"): 1711 userdir = os.path.expanduser(homedir) 1712 if os.path.exists(userdir) is True: 1713 homedir = os.path.realpath(userdir) 1714 else: 1715 homedir = None 1716else: 1717 homedir = os.path.realpath(homedir) 1718 1719if os.path.exists(homedir) is False: 1720 homedir = None 1721else: 1722 if os.path.isdir(homedir) is False: 1723 homedir = None 1724 else: 1725 pass 1726 1727if homedir is not None: 1728 c.home_dir = homedir 1729else: 1730 pass 1731 1732try: 1733 result = c.key_export_secret(pattern=logrus) 1734except: 1735 result = c.key_export_secret(pattern=None) 1736 1737if result is not None: 1738 with open(keyfile, "wb") as f: 1739 f.write(result) 1740 os.chmod(keyfile, 0o600) 1741else: 1742 pass 1743#+END_SRC 1744 1745Alternatively the approach of the following script can be used. This 1746longer example saves the exported secret key(s) in files in the GnuPG 1747home directory, in addition to setting the file permissions as only 1748readable and writable by the user. It also exports the secret key(s) 1749twice in order to output both GPG binary (=.gpg=) and ASCII armoured 1750(=.asc=) files. 1751 1752#+BEGIN_SRC python -i 1753import gpg 1754import os 1755import os.path 1756import subprocess 1757import sys 1758 1759print(""" 1760This script exports one or more secret keys as both ASCII armored and binary 1761file formats, saved in files within the user's GPG home directory. 1762 1763The gpg-agent and pinentry are invoked to authorise the export. 1764""") 1765 1766if sys.platform == "win32": 1767 gpgconfcmd = "gpgconf.exe --list-dirs homedir" 1768else: 1769 gpgconfcmd = "gpgconf --list-dirs homedir" 1770 1771a = gpg.Context(armor=True) 1772b = gpg.Context() 1773c = gpg.Context() 1774 1775if len(sys.argv) >= 4: 1776 keyfile = sys.argv[1] 1777 logrus = sys.argv[2] 1778 homedir = sys.argv[3] 1779elif len(sys.argv) == 3: 1780 keyfile = sys.argv[1] 1781 logrus = sys.argv[2] 1782 homedir = input("Enter the GPG configuration directory path (optional): ") 1783elif len(sys.argv) == 2: 1784 keyfile = sys.argv[1] 1785 logrus = input("Enter the UID matching the secret key(s) to export: ") 1786 homedir = input("Enter the GPG configuration directory path (optional): ") 1787else: 1788 keyfile = input("Enter the filename to save the secret key to: ") 1789 logrus = input("Enter the UID matching the secret key(s) to export: ") 1790 homedir = input("Enter the GPG configuration directory path (optional): ") 1791 1792if len(homedir) == 0: 1793 homedir = None 1794elif homedir.startswith("~"): 1795 userdir = os.path.expanduser(homedir) 1796 if os.path.exists(userdir) is True: 1797 homedir = os.path.realpath(userdir) 1798 else: 1799 homedir = None 1800else: 1801 homedir = os.path.realpath(homedir) 1802 1803if os.path.exists(homedir) is False: 1804 homedir = None 1805else: 1806 if os.path.isdir(homedir) is False: 1807 homedir = None 1808 else: 1809 pass 1810 1811if homedir is not None: 1812 c.home_dir = homedir 1813else: 1814 pass 1815 1816if c.home_dir is not None: 1817 if c.home_dir.endswith("/"): 1818 gpgfile = "{0}{1}.gpg".format(c.home_dir, keyfile) 1819 ascfile = "{0}{1}.asc".format(c.home_dir, keyfile) 1820 else: 1821 gpgfile = "{0}/{1}.gpg".format(c.home_dir, keyfile) 1822 ascfile = "{0}/{1}.asc".format(c.home_dir, keyfile) 1823else: 1824 if os.path.exists(os.environ["GNUPGHOME"]) is True: 1825 hd = os.environ["GNUPGHOME"] 1826 else: 1827 try: 1828 hd = subprocess.getoutput(gpgconfcmd) 1829 except: 1830 process = subprocess.Popen(gpgconfcmd.split(), 1831 stdout=subprocess.PIPE) 1832 procom = process.communicate() 1833 if sys.version_info[0] == 2: 1834 hd = procom[0].strip() 1835 else: 1836 hd = procom[0].decode().strip() 1837 gpgfile = "{0}/{1}.gpg".format(hd, keyfile) 1838 ascfile = "{0}/{1}.asc".format(hd, keyfile) 1839 1840try: 1841 a_result = a.key_export_secret(pattern=logrus) 1842 b_result = b.key_export_secret(pattern=logrus) 1843except: 1844 a_result = a.key_export_secret(pattern=None) 1845 b_result = b.key_export_secret(pattern=None) 1846 1847if a_result is not None: 1848 with open(ascfile, "wb") as f: 1849 f.write(a_result) 1850 os.chmod(ascfile, 0o600) 1851else: 1852 pass 1853 1854if b_result is not None: 1855 with open(gpgfile, "wb") as f: 1856 f.write(b_result) 1857 os.chmod(gpgfile, 0o600) 1858else: 1859 pass 1860#+END_SRC 1861 1862 1863*** Sending public keys to the SKS Keyservers 1864 :PROPERTIES: 1865 :CUSTOM_ID: howto-send-public-key 1866 :END: 1867 1868As with the previous section on importing keys, the =hkp4py= module 1869adds another option with exporting keys in order to send them to the 1870public keyservers. 1871 1872The following example demonstrates how this may be done. 1873 1874#+BEGIN_SRC python -i 1875import gpg 1876import hkp4py 1877import os.path 1878import sys 1879 1880print(""" 1881This script sends one or more public keys to the SKS keyservers and is 1882essentially a slight variation on the export-key.py script. 1883""") 1884 1885c = gpg.Context(armor=True) 1886server = hkp4py.KeyServer("hkps://hkps.pool.sks-keyservers.net") 1887 1888if len(sys.argv) > 2: 1889 logrus = " ".join(sys.argv[1:]) 1890elif len(sys.argv) == 2: 1891 logrus = sys.argv[1] 1892else: 1893 logrus = input("Enter the UID matching the key(s) to send: ") 1894 1895if len(logrus) > 0: 1896 try: 1897 export_result = c.key_export(pattern=logrus) 1898 except Exception as e: 1899 print(e) 1900 export_result = None 1901else: 1902 export_result = c.key_export(pattern=None) 1903 1904if export_result is not None: 1905 try: 1906 try: 1907 send_result = server.add(export_result) 1908 except: 1909 send_result = server.add(export_result.decode()) 1910 if send_result is not None: 1911 print(send_result) 1912 else: 1913 pass 1914 except Exception as e: 1915 print(e) 1916else: 1917 pass 1918#+END_SRC 1919 1920An expanded version of this script with additional functions for 1921specifying an alternative homedir location is in the examples 1922directory as =send-key-to-keyserver.py=. 1923 1924The =hkp4py= module appears to handle both string and byte literal text 1925data equally well, but the GPGME bindings deal primarily with byte 1926literal data only and so this script sends in that format first, then 1927tries the string literal form. 1928 1929 1930* Basic Functions 1931 :PROPERTIES: 1932 :CUSTOM_ID: howto-the-basics 1933 :END: 1934 1935The most frequently called features of any cryptographic library will 1936be the most fundamental tasks for encryption software. In this 1937section we will look at how to programmatically encrypt data, decrypt 1938it, sign it and verify signatures. 1939 1940 1941** Encryption 1942 :PROPERTIES: 1943 :CUSTOM_ID: howto-basic-encryption 1944 :END: 1945 1946Encrypting is very straight forward. In the first example below the 1947message, =text=, is encrypted to a single recipient's key. In the 1948second example the message will be encrypted to multiple recipients. 1949 1950 1951*** Encrypting to one key 1952 :PROPERTIES: 1953 :CUSTOM_ID: howto-basic-encryption-single 1954 :END: 1955 1956Once the the Context is set the main issues with encrypting data is 1957essentially reduced to key selection and the keyword arguments 1958specified in the =gpg.Context().encrypt()= method. 1959 1960Those keyword arguments are: =recipients=, a list of keys encrypted to 1961(covered in greater detail in the following section); =sign=, whether 1962or not to sign the plaintext data, see subsequent sections on signing 1963and verifying signatures below (defaults to =True=); =sink=, to write 1964results or partial results to a secure sink instead of returning it 1965(defaults to =None=); =passphrase=, only used when utilising symmetric 1966encryption (defaults to =None=); =always_trust=, used to override the 1967trust model settings for recipient keys (defaults to =False=); 1968=add_encrypt_to=, utilises any preconfigured =encrypt-to= or 1969=default-key= settings in the user's =gpg.conf= file (defaults to 1970=False=); =prepare=, prepare for encryption (defaults to =False=); 1971=expect_sign=, prepare for signing (defaults to =False=); =compress=, 1972compresses the plaintext prior to encryption (defaults to =True=). 1973 1974#+BEGIN_SRC python -i 1975import gpg 1976 1977a_key = "0x12345678DEADBEEF" 1978text = b"""Some text to test with. 1979 1980Since the text in this case must be bytes, it is most likely that 1981the input form will be a separate file which is opened with "rb" 1982as this is the simplest method of obtaining the correct data format. 1983""" 1984 1985c = gpg.Context(armor=True) 1986rkey = list(c.keylist(pattern=a_key, secret=False)) 1987ciphertext, result, sign_result = c.encrypt(text, recipients=rkey, sign=False) 1988 1989with open("secret_plans.txt.asc", "wb") as afile: 1990 afile.write(ciphertext) 1991#+END_SRC 1992 1993Though this is even more likely to be used like this; with the 1994plaintext input read from a file, the recipient keys used for 1995encryption regardless of key trust status and the encrypted output 1996also encrypted to any preconfigured keys set in the =gpg.conf= file: 1997 1998#+BEGIN_SRC python -i 1999import gpg 2000 2001a_key = "0x12345678DEADBEEF" 2002 2003with open("secret_plans.txt", "rb") as afile: 2004 text = afile.read() 2005 2006c = gpg.Context(armor=True) 2007rkey = list(c.keylist(pattern=a_key, secret=False)) 2008ciphertext, result, sign_result = c.encrypt(text, recipients=rkey, sign=True, 2009 always_trust=True, 2010 add_encrypt_to=True) 2011 2012with open("secret_plans.txt.asc", "wb") as afile: 2013 afile.write(ciphertext) 2014#+END_SRC 2015 2016If the =recipients= parameter is empty then the plaintext is encrypted 2017symmetrically. If no =passphrase= is supplied as a parameter or via a 2018callback registered with the =Context()= then an out-of-band prompt 2019for the passphrase via pinentry will be invoked. 2020 2021 2022*** Encrypting to multiple keys 2023 :PROPERTIES: 2024 :CUSTOM_ID: howto-basic-encryption-multiple 2025 :END: 2026 2027Encrypting to multiple keys essentially just expands upon the key 2028selection process and the recipients from the previous examples. 2029 2030The following example encrypts a message (=text=) to everyone with an 2031email address on the =gnupg.org= domain,[fn:4] but does /not/ encrypt 2032to a default key or other key which is configured to normally encrypt 2033to. 2034 2035#+BEGIN_SRC python -i 2036import gpg 2037 2038text = b"""Oh look, another test message. 2039 2040The same rules apply as with the previous example and more likely 2041than not, the message will actually be drawn from reading the 2042contents of a file or, maybe, from entering data at an input() 2043prompt. 2044 2045Since the text in this case must be bytes, it is most likely that 2046the input form will be a separate file which is opened with "rb" 2047as this is the simplest method of obtaining the correct data 2048format. 2049""" 2050 2051c = gpg.Context(armor=True) 2052rpattern = list(c.keylist(pattern="@gnupg.org", secret=False)) 2053logrus = [] 2054 2055for i in range(len(rpattern)): 2056 if rpattern[i].can_encrypt == 1: 2057 logrus.append(rpattern[i]) 2058 2059ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, 2060 sign=False, always_trust=True) 2061 2062with open("secret_plans.txt.asc", "wb") as afile: 2063 afile.write(ciphertext) 2064#+END_SRC 2065 2066All it would take to change the above example to sign the message 2067and also encrypt the message to any configured default keys would 2068be to change the =c.encrypt= line to this: 2069 2070#+BEGIN_SRC python -i 2071ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, 2072 always_trust=True, 2073 add_encrypt_to=True) 2074#+END_SRC 2075 2076The only keyword arguments requiring modification are those for which 2077the default values are changing. The default value of =sign= is 2078=True=, the default of =always_trust= is =False=, the default of 2079=add_encrypt_to= is =False=. 2080 2081If =always_trust= is not set to =True= and any of the recipient keys 2082are not trusted (e.g. not signed or locally signed) then the 2083encryption will raise an error. It is possible to mitigate this 2084somewhat with something more like this: 2085 2086#+BEGIN_SRC python -i 2087import gpg 2088 2089with open("secret_plans.txt.asc", "rb") as afile: 2090 text = afile.read() 2091 2092c = gpg.Context(armor=True) 2093rpattern = list(c.keylist(pattern="@gnupg.org", secret=False)) 2094logrus = [] 2095 2096for i in range(len(rpattern)): 2097 if rpattern[i].can_encrypt == 1: 2098 logrus.append(rpattern[i]) 2099 2100 try: 2101 ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, 2102 add_encrypt_to=True) 2103 except gpg.errors.InvalidRecipients as e: 2104 for i in range(len(e.recipients)): 2105 for n in range(len(logrus)): 2106 if logrus[n].fpr == e.recipients[i].fpr: 2107 logrus.remove(logrus[n]) 2108 else: 2109 pass 2110 try: 2111 ciphertext, result, sign_result = c.encrypt(text, 2112 recipients=logrus, 2113 add_encrypt_to=True) 2114 with open("secret_plans.txt.asc", "wb") as afile: 2115 afile.write(ciphertext) 2116 except: 2117 pass 2118#+END_SRC 2119 2120This will attempt to encrypt to all the keys searched for, then remove 2121invalid recipients if it fails and try again. 2122 2123 2124** Decryption 2125 :PROPERTIES: 2126 :CUSTOM_ID: howto-basic-decryption 2127 :END: 2128 2129Decrypting something encrypted to a key in one's secret keyring is 2130fairly straight forward. 2131 2132In this example code, however, preconfiguring either =gpg.Context()= 2133or =gpg.core.Context()= as =c= is unnecessary because there is no need 2134to modify the Context prior to conducting the decryption and since the 2135Context is only used once, setting it to =c= simply adds lines for no 2136gain. 2137 2138#+BEGIN_SRC python -i 2139import gpg 2140 2141ciphertext = input("Enter path and filename of encrypted file: ") 2142newfile = input("Enter path and filename of file to save decrypted data to: ") 2143 2144with open(ciphertext, "rb") as cfile: 2145 try: 2146 plaintext, result, verify_result = gpg.Context().decrypt(cfile) 2147 except gpg.errors.GPGMEError as e: 2148 plaintext = None 2149 print(e) 2150 2151if plaintext is not None: 2152 with open(newfile, "wb") as nfile: 2153 nfile.write(plaintext) 2154 else: 2155 pass 2156#+END_SRC 2157 2158The data available in =plaintext= in this example is the decrypted 2159content as a byte object, the recipient key IDs and algorithms in 2160=result= and the results of verifying any signatures of the data in 2161=verify_result=. 2162 2163If =gpg.Context().decrypt(cfile, verify=False)= is called instead, 2164then =verify_result= will be returned as =None= and the rest remains 2165as described here. 2166 2167 2168** Signing text and files 2169 :PROPERTIES: 2170 :CUSTOM_ID: howto-basic-signing 2171 :END: 2172 2173The following sections demonstrate how to specify keys to sign with. 2174 2175 2176*** Signing key selection 2177 :PROPERTIES: 2178 :CUSTOM_ID: howto-basic-signing-signers 2179 :END: 2180 2181By default GPGME and the Python bindings will use the default key 2182configured for the user invoking the GPGME API. If there is no 2183default key specified and there is more than one secret key available 2184it may be necessary to specify the key or keys with which to sign 2185messages and files. 2186 2187#+BEGIN_SRC python -i 2188import gpg 2189 2190logrus = input("Enter the email address or string to match signing keys to: ") 2191hancock = gpg.Context().keylist(pattern=logrus, secret=True) 2192sig_src = list(hancock) 2193#+END_SRC 2194 2195The signing examples in the following sections include the explicitly 2196designated =signers= parameter in two of the five examples; once where 2197the resulting signature would be ASCII armoured and once where it 2198would not be armoured. 2199 2200While it would be possible to enter a key ID or fingerprint here to 2201match a specific key, it is not possible to enter two fingerprints and 2202match two keys since the patten expects a string, bytes or None and 2203not a list. A string with two fingerprints won't match any single 2204key. 2205 2206 2207*** Normal or default signing messages or files 2208 :PROPERTIES: 2209 :CUSTOM_ID: howto-basic-signing-normal 2210 :END: 2211 2212The normal or default signing process is essentially the same as is 2213most often invoked when also encrypting a message or file. So when 2214the encryption component is not utilised, the result is to produce an 2215encoded and signed output which may or may not be ASCII armoured and 2216which may or may not also be compressed. 2217 2218By default compression will be used unless GnuPG detects that the 2219plaintext is already compressed. ASCII armouring will be determined 2220according to the value of =gpg.Context().armor=. 2221 2222The compression algorithm is selected in much the same way as the 2223symmetric encryption algorithm or the hash digest algorithm is when 2224multiple keys are involved; from the preferences saved into the key 2225itself or by comparison with the preferences with all other keys 2226involved. 2227 2228#+BEGIN_SRC python -i 2229import gpg 2230 2231text0 = """Declaration of ... something. 2232 2233""" 2234text = text0.encode() 2235 2236c = gpg.Context(armor=True, signers=sig_src) 2237signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.NORMAL) 2238 2239with open("/path/to/statement.txt.asc", "w") as afile: 2240 afile.write(signed_data.decode()) 2241#+END_SRC 2242 2243Though everything in this example is accurate, it is more likely that 2244reading the input data from another file and writing the result to a 2245new file will be performed more like the way it is done in the next 2246example. Even if the output format is ASCII armoured. 2247 2248#+BEGIN_SRC python -i 2249import gpg 2250 2251with open("/path/to/statement.txt", "rb") as tfile: 2252 text = tfile.read() 2253 2254c = gpg.Context() 2255signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.NORMAL) 2256 2257with open("/path/to/statement.txt.sig", "wb") as afile: 2258 afile.write(signed_data) 2259#+END_SRC 2260 2261 2262*** Detached signing messages and files 2263 :PROPERTIES: 2264 :CUSTOM_ID: howto-basic-signing-detached 2265 :END: 2266 2267Detached signatures will often be needed in programmatic uses of 2268GPGME, either for signing files (e.g. tarballs of code releases) or as 2269a component of message signing (e.g. PGP/MIME encoded email). 2270 2271#+BEGIN_SRC python -i 2272import gpg 2273 2274text0 = """Declaration of ... something. 2275 2276""" 2277text = text0.encode() 2278 2279c = gpg.Context(armor=True) 2280signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.DETACH) 2281 2282with open("/path/to/statement.txt.asc", "w") as afile: 2283 afile.write(signed_data.decode()) 2284#+END_SRC 2285 2286As with normal signatures, detached signatures are best handled as 2287byte literals, even when the output is ASCII armoured. 2288 2289#+BEGIN_SRC python -i 2290import gpg 2291 2292with open("/path/to/statement.txt", "rb") as tfile: 2293 text = tfile.read() 2294 2295c = gpg.Context(signers=sig_src) 2296signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.DETACH) 2297 2298with open("/path/to/statement.txt.sig", "wb") as afile: 2299 afile.write(signed_data) 2300#+END_SRC 2301 2302 2303*** Clearsigning messages or text 2304 :PROPERTIES: 2305 :CUSTOM_ID: howto-basic-signing-clear 2306 :END: 2307 2308Though PGP/in-line messages are no longer encouraged in favour of 2309PGP/MIME, there is still sometimes value in utilising in-line 2310signatures. This is where clear-signed messages or text is of value. 2311 2312#+BEGIN_SRC python -i 2313import gpg 2314 2315text0 = """Declaration of ... something. 2316 2317""" 2318text = text0.encode() 2319 2320c = gpg.Context() 2321signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.CLEAR) 2322 2323with open("/path/to/statement.txt.asc", "w") as afile: 2324 afile.write(signed_data.decode()) 2325#+END_SRC 2326 2327In spite of the appearance of a clear-signed message, the data handled 2328by GPGME in signing it must still be byte literals. 2329 2330#+BEGIN_SRC python -i 2331import gpg 2332 2333with open("/path/to/statement.txt", "rb") as tfile: 2334 text = tfile.read() 2335 2336c = gpg.Context() 2337signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.CLEAR) 2338 2339with open("/path/to/statement.txt.asc", "wb") as afile: 2340 afile.write(signed_data) 2341#+END_SRC 2342 2343 2344** Signature verification 2345 :PROPERTIES: 2346 :CUSTOM_ID: howto-basic-verification 2347 :END: 2348 2349Essentially there are two principal methods of verification of a 2350signature. The first of these is for use with the normal or default 2351signing method and for clear-signed messages. The second is for use 2352with files and data with detached signatures. 2353 2354The following example is intended for use with the default signing 2355method where the file was not ASCII armoured: 2356 2357#+BEGIN_SRC python -i 2358import gpg 2359import time 2360 2361filename = "statement.txt" 2362gpg_file = "statement.txt.gpg" 2363 2364c = gpg.Context() 2365 2366try: 2367 data, result = c.verify(open(gpg_file)) 2368 verified = True 2369except gpg.errors.BadSignatures as e: 2370 verified = False 2371 print(e) 2372 2373if verified is True: 2374 for i in range(len(result.signatures)): 2375 sign = result.signatures[i] 2376 print("""Good signature from: 2377{0} 2378with key {1} 2379made at {2} 2380""".format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, 2381 time.ctime(sign.timestamp))) 2382else: 2383 pass 2384#+END_SRC 2385 2386Whereas this next example, which is almost identical would work with 2387normal ASCII armoured files and with clear-signed files: 2388 2389#+BEGIN_SRC python -i 2390import gpg 2391import time 2392 2393filename = "statement.txt" 2394asc_file = "statement.txt.asc" 2395 2396c = gpg.Context() 2397 2398try: 2399 data, result = c.verify(open(asc_file)) 2400 verified = True 2401except gpg.errors.BadSignatures as e: 2402 verified = False 2403 print(e) 2404 2405if verified is True: 2406 for i in range(len(result.signatures)): 2407 sign = result.signatures[i] 2408 print("""Good signature from: 2409{0} 2410with key {1} 2411made at {2} 2412""".format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, 2413 time.ctime(sign.timestamp))) 2414else: 2415 pass 2416#+END_SRC 2417 2418In both of the previous examples it is also possible to compare the 2419original data that was signed against the signed data in =data= to see 2420if it matches with something like this: 2421 2422#+BEGIN_SRC python -i 2423with open(filename, "rb") as afile: 2424 text = afile.read() 2425 2426if text == data: 2427 print("Good signature.") 2428else: 2429 pass 2430#+END_SRC 2431 2432The following two examples, however, deal with detached signatures. 2433With his method of verification the data that was signed does not get 2434returned since it is already being explicitly referenced in the first 2435argument of =c.verify=. So =data= is =None= and only the information 2436in =result= is available. 2437 2438#+BEGIN_SRC python -i 2439import gpg 2440import time 2441 2442filename = "statement.txt" 2443sig_file = "statement.txt.sig" 2444 2445c = gpg.Context() 2446 2447try: 2448 data, result = c.verify(open(filename), open(sig_file)) 2449 verified = True 2450except gpg.errors.BadSignatures as e: 2451 verified = False 2452 print(e) 2453 2454if verified is True: 2455 for i in range(len(result.signatures)): 2456 sign = result.signatures[i] 2457 print("""Good signature from: 2458{0} 2459with key {1} 2460made at {2} 2461""".format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, 2462 time.ctime(sign.timestamp))) 2463else: 2464 pass 2465#+END_SRC 2466 2467#+BEGIN_SRC python -i 2468import gpg 2469import time 2470 2471filename = "statement.txt" 2472asc_file = "statement.txt.asc" 2473 2474c = gpg.Context() 2475 2476try: 2477 data, result = c.verify(open(filename), open(asc_file)) 2478 verified = True 2479except gpg.errors.BadSignatures as e: 2480 verified = False 2481 print(e) 2482 2483if verified is True: 2484 for i in range(len(result.signatures)): 2485 sign = result.signatures[i] 2486 print("""Good signature from: 2487{0} 2488with key {1} 2489made at {2} 2490""".format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, 2491 time.ctime(sign.timestamp))) 2492else: 2493 pass 2494#+END_SRC 2495 2496 2497* Creating keys and subkeys 2498 :PROPERTIES: 2499 :CUSTOM_ID: key-generation 2500 :END: 2501 2502The one thing, aside from GnuPG itself, that GPGME depends on, of 2503course, is the keys themselves. So it is necessary to be able to 2504generate them and modify them by adding subkeys, revoking or disabling 2505them, sometimes deleting them and doing the same for user IDs. 2506 2507In the following examples a key will be created for the world's 2508greatest secret agent, Danger Mouse. Since Danger Mouse is a secret 2509agent he needs to be able to protect information to =SECRET= level 2510clearance, so his keys will be 3072-bit keys. 2511 2512The pre-configured =gpg.conf= file which sets cipher, digest and other 2513preferences contains the following configuration parameters: 2514 2515#+BEGIN_SRC conf 2516 expert 2517 allow-freeform-uid 2518 allow-secret-key-import 2519 trust-model tofu+pgp 2520 tofu-default-policy unknown 2521 enable-large-rsa 2522 enable-dsa2 2523 cert-digest-algo SHA512 2524 default-preference-list TWOFISH CAMELLIA256 AES256 CAMELLIA192 AES192 CAMELLIA128 AES BLOWFISH IDEA CAST5 3DES SHA512 SHA384 SHA256 SHA224 RIPEMD160 SHA1 ZLIB BZIP2 ZIP Uncompressed 2525 personal-cipher-preferences TWOFISH CAMELLIA256 AES256 CAMELLIA192 AES192 CAMELLIA128 AES BLOWFISH IDEA CAST5 3DES 2526 personal-digest-preferences SHA512 SHA384 SHA256 SHA224 RIPEMD160 SHA1 2527 personal-compress-preferences ZLIB BZIP2 ZIP Uncompressed 2528#+END_SRC 2529 2530 2531** Primary key 2532 :PROPERTIES: 2533 :CUSTOM_ID: keygen-primary 2534 :END: 2535 2536Generating a primary key uses the =create_key= method in a Context. 2537It contains multiple arguments and keyword arguments, including: 2538=userid=, =algorithm=, =expires_in=, =expires=, =sign=, =encrypt=, 2539=certify=, =authenticate=, =passphrase= and =force=. The defaults for 2540all of those except =userid=, =algorithm=, =expires_in=, =expires= and 2541=passphrase= is =False=. The defaults for =algorithm= and 2542=passphrase= is =None=. The default for =expires_in= is =0=. The 2543default for =expires= is =True=. There is no default for =userid=. 2544 2545If =passphrase= is left as =None= then the key will not be generated 2546with a passphrase, if =passphrase= is set to a string then that will 2547be the passphrase and if =passphrase= is set to =True= then gpg-agent 2548will launch pinentry to prompt for a passphrase. For the sake of 2549convenience, these examples will keep =passphrase= set to =None=. 2550 2551#+BEGIN_SRC python -i 2552import gpg 2553 2554c = gpg.Context() 2555 2556c.home_dir = "~/.gnupg-dm" 2557userid = "Danger Mouse <dm@secret.example.net>" 2558 2559dmkey = c.create_key(userid, algorithm="rsa3072", expires_in=31536000, 2560 sign=True, certify=True) 2561#+END_SRC 2562 2563One thing to note here is the use of setting the =c.home_dir= 2564parameter. This enables generating the key or keys in a different 2565location. In this case to keep the new key data created for this 2566example in a separate location rather than adding it to existing and 2567active key store data. As with the default directory, =~/.gnupg=, any 2568temporary or separate directory needs the permissions set to only 2569permit access by the directory owner. On posix systems this means 2570setting the directory permissions to 700. 2571 2572The =temp-homedir-config.py= script in the HOWTO examples directory 2573will create an alternative homedir with these configuration options 2574already set and the correct directory and file permissions. 2575 2576The successful generation of the key can be confirmed via the returned 2577=GenkeyResult= object, which includes the following data: 2578 2579#+BEGIN_SRC python -i 2580print(""" 2581 Fingerprint: {0} 2582 Primary Key: {1} 2583 Public Key: {2} 2584 Secret Key: {3} 2585 Sub Key: {4} 2586User IDs: {5} 2587""".format(dmkey.fpr, dmkey.primary, dmkey.pubkey, dmkey.seckey, dmkey.sub, 2588 dmkey.uid)) 2589#+END_SRC 2590 2591Alternatively the information can be confirmed using the command line 2592program: 2593 2594#+BEGIN_SRC shell 2595 bash-4.4$ gpg --homedir ~/.gnupg-dm -K 2596 ~/.gnupg-dm/pubring.kbx 2597 ---------------------- 2598 sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] 2599 177B7C25DB99745EE2EE13ED026D2F19E99E63AA 2600 uid [ultimate] Danger Mouse <dm@secret.example.net> 2601 2602 bash-4.4$ 2603#+END_SRC 2604 2605As with generating keys manually, to preconfigure expanded preferences 2606for the cipher, digest and compression algorithms, the =gpg.conf= file 2607must contain those details in the home directory in which the new key 2608is being generated. I used a cut down version of my own =gpg.conf= 2609file in order to be able to generate this: 2610 2611#+BEGIN_SRC shell 2612 bash-4.4$ gpg --homedir ~/.gnupg-dm --edit-key 177B7C25DB99745EE2EE13ED026D2F19E99E63AA showpref quit 2613 Secret key is available. 2614 2615 sec rsa3072/026D2F19E99E63AA 2616 created: 2018-03-15 expires: 2019-03-15 usage: SC 2617 trust: ultimate validity: ultimate 2618 [ultimate] (1). Danger Mouse <dm@secret.example.net> 2619 2620 [ultimate] (1). Danger Mouse <dm@secret.example.net> 2621 Cipher: TWOFISH, CAMELLIA256, AES256, CAMELLIA192, AES192, CAMELLIA128, AES, BLOWFISH, IDEA, CAST5, 3DES 2622 Digest: SHA512, SHA384, SHA256, SHA224, RIPEMD160, SHA1 2623 Compression: ZLIB, BZIP2, ZIP, Uncompressed 2624 Features: MDC, Keyserver no-modify 2625 2626 bash-4.4$ 2627#+END_SRC 2628 2629 2630** Subkeys 2631 :PROPERTIES: 2632 :CUSTOM_ID: keygen-subkeys 2633 :END: 2634 2635Adding subkeys to a primary key is fairly similar to creating the 2636primary key with the =create_subkey= method. Most of the arguments 2637are the same, but not quite all. Instead of the =userid= argument 2638there is now a =key= argument for selecting which primary key to add 2639the subkey to. 2640 2641In the following example an encryption subkey will be added to the 2642primary key. Since Danger Mouse is a security conscious secret agent, 2643this subkey will only be valid for about six months, half the length 2644of the primary key. 2645 2646#+BEGIN_SRC python -i 2647import gpg 2648 2649c = gpg.Context() 2650c.home_dir = "~/.gnupg-dm" 2651 2652key = c.get_key(dmkey.fpr, secret=True) 2653dmsub = c.create_subkey(key, algorithm="rsa3072", expires_in=15768000, 2654 encrypt=True) 2655#+END_SRC 2656 2657As with the primary key, the results here can be checked with: 2658 2659#+BEGIN_SRC python -i 2660print(""" 2661 Fingerprint: {0} 2662 Primary Key: {1} 2663 Public Key: {2} 2664 Secret Key: {3} 2665 Sub Key: {4} 2666User IDs: {5} 2667""".format(dmsub.fpr, dmsub.primary, dmsub.pubkey, dmsub.seckey, dmsub.sub, 2668 dmsub.uid)) 2669#+END_SRC 2670 2671As well as on the command line with: 2672 2673#+BEGIN_SRC shell 2674 bash-4.4$ gpg --homedir ~/.gnupg-dm -K 2675 ~/.gnupg-dm/pubring.kbx 2676 ---------------------- 2677 sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] 2678 177B7C25DB99745EE2EE13ED026D2F19E99E63AA 2679 uid [ultimate] Danger Mouse <dm@secret.example.net> 2680 ssb rsa3072 2018-03-15 [E] [expires: 2018-09-13] 2681 2682 bash-4.4$ 2683#+END_SRC 2684 2685 2686** User IDs 2687 :PROPERTIES: 2688 :CUSTOM_ID: keygen-uids 2689 :END: 2690 2691 2692*** Adding User IDs 2693 :PROPERTIES: 2694 :CUSTOM_ID: keygen-uids-add 2695 :END: 2696 2697By comparison to creating primary keys and subkeys, adding a new user 2698ID to an existing key is much simpler. The method used to do this is 2699=key_add_uid= and the only arguments it takes are for the =key= and 2700the new =uid=. 2701 2702#+BEGIN_SRC python -i 2703import gpg 2704 2705c = gpg.Context() 2706c.home_dir = "~/.gnupg-dm" 2707 2708dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" 2709key = c.get_key(dmfpr, secret=True) 2710uid = "Danger Mouse <danger.mouse@secret.example.net>" 2711 2712c.key_add_uid(key, uid) 2713#+END_SRC 2714 2715Unsurprisingly the result of this is: 2716 2717#+BEGIN_SRC shell 2718 bash-4.4$ gpg --homedir ~/.gnupg-dm -K 2719 ~/.gnupg-dm/pubring.kbx 2720 ---------------------- 2721 sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] 2722 177B7C25DB99745EE2EE13ED026D2F19E99E63AA 2723 uid [ultimate] Danger Mouse <danger.mouse@secret.example.net> 2724 uid [ultimate] Danger Mouse <dm@secret.example.net> 2725 ssb rsa3072 2018-03-15 [E] [expires: 2018-09-13] 2726 2727 bash-4.4$ 2728#+END_SRC 2729 2730 2731*** Revoking User IDs 2732 :PROPERTIES: 2733 :CUSTOM_ID: keygen-uids-revoke 2734 :END: 2735 2736Revoking a user ID is a fairly similar process, except that it uses 2737the =key_revoke_uid= method. 2738 2739#+BEGIN_SRC python -i 2740import gpg 2741 2742c = gpg.Context() 2743c.home_dir = "~/.gnupg-dm" 2744 2745dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" 2746key = c.get_key(dmfpr, secret=True) 2747uid = "Danger Mouse <danger.mouse@secret.example.net>" 2748 2749c.key_revoke_uid(key, uid) 2750#+END_SRC 2751 2752 2753** Key certification 2754 :PROPERTIES: 2755 :CUSTOM_ID: key-sign 2756 :END: 2757 2758Since key certification is more frequently referred to as key signing, 2759the method used to perform this function is =key_sign=. 2760 2761The =key_sign= method takes four arguments: =key=, =uids=, 2762=expires_in= and =local=. The default value of =uids= is =None= and 2763which results in all user IDs being selected. The default value of 2764both =expires_in= and =local= is =False=; which results in the 2765signature never expiring and being able to be exported. 2766 2767The =key= is the key being signed rather than the key doing the 2768signing. To change the key doing the signing refer to the signing key 2769selection above for signing messages and files. 2770 2771If the =uids= value is not =None= then it must either be a string to 2772match a single user ID or a list of strings to match multiple user 2773IDs. In this case the matching of those strings must be precise and 2774it is case sensitive. 2775 2776To sign Danger Mouse's key for just the initial user ID with a 2777signature which will last a little over a month, do this: 2778 2779#+BEGIN_SRC python -i 2780import gpg 2781 2782c = gpg.Context() 2783uid = "Danger Mouse <dm@secret.example.net>" 2784 2785dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" 2786key = c.get_key(dmfpr, secret=True) 2787c.key_sign(key, uids=uid, expires_in=2764800) 2788#+END_SRC 2789 2790 2791*** Verifying key certifications 2792 :PROPERTIES: 2793 :CUSTOM_ID: key-sign-verify 2794 :END: 2795 2796#+BEGIN_SRC python -i 2797import gpg 2798import time 2799 2800c = gpg.Context() 2801dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" 2802keys = list(c.keylist(pattern=dmuid, mode=gpg.constants.keylist.mode.SIGS)) 2803key = keys[0] 2804 2805for user in key.uids: 2806 for sig in user.signatures: 2807 print("0x{0}".format(sig.keyid), "", time.ctime(sig.timestamp), "", 2808 sig.uid) 2809#+END_SRC 2810 2811Which for Danger Mouse displays the following: 2812 2813#+BEGIN_EXAMPLE 2814 0x92E3F6115435C65A Thu Mar 15 13:17:44 2018 Danger Mouse <dm@secret.example.net> 2815 0x321E4E2373590E5D Mon Nov 26 12:46:05 2018 Ben McGinnes <ben@adversary.org> 2816#+END_EXAMPLE 2817 2818The two key signatures listed are for the self-certification of Danger 2819Mouse's key made when the key was created in March, 2018; and the 2820second is a signature made by the author and set to expire at the end 2821of the year. Note that the second signature was made with the 2822following code (including the preceding code to display the output of 2823the certifications or key signatures): 2824 2825#+BEGIN_SRC python -i 2826import gpg 2827import math 2828import pendulum 2829import time 2830 2831hd = "/home/dm/.gnupg" 2832c = gpg.Context() 2833d = gpg.Context(home_dir=hd) 2834dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" 2835dmuid = "Danger Mouse <dm@secret.example.net>" 2836dkeys = list(c.keylist(pattern=dmuid)) 2837dmkey = dkeys[0] 2838 2839c.key_import(d.key_export(pattern=None)) 2840 2841tp = pendulum.period(pendulum.now(tz="local"), pendulum.datetime(2019, 1, 1)) 2842ts = tp.total_seconds() 2843total_secs = math.ceil(ts) 2844c.key_sign(dmkey, uids=dmuid, expires_in=total_secs) 2845 2846d.key_import(c.key_export(pattern=dmuid)) 2847keys = list(c.keylist(pattern=dmuid, mode=gpg.constants.keylist.mode.SIGS)) 2848key = keys[0] 2849 2850for user in key.uids: 2851 for sig in user.signatures: 2852 print("0x{0}".format(sig.keyid), "", time.ctime(sig.timestamp), "", 2853 sig.uid) 2854#+END_SRC 2855 2856Note that this final code block includes the use of a module which is 2857/not/ part of Python's standard library, the [[https://pendulum.eustace.io/][pendulum module]]. Unlike 2858the standard datetime module, pendulum makes working with dates and 2859times significantly easier in Python; just as the requests module 2860makes working with HTTP and HTTPS easier than the builtin modules do. 2861 2862Though neither requests nor pendulum are required modules for using 2863the GPGME Python bindings, they are both highly recommended more 2864generally. 2865 2866 2867* Advanced or Experimental Use Cases 2868 :PROPERTIES: 2869 :CUSTOM_ID: advanced-use 2870 :END: 2871 2872 2873** C plus Python plus SWIG plus Cython 2874 :PROPERTIES: 2875 :CUSTOM_ID: cython 2876 :END: 2877 2878In spite of the apparent incongruence of using Python bindings to a C 2879interface only to generate more C from the Python; it is in fact quite 2880possible to use the GPGME bindings with [[http://docs.cython.org/en/latest/index.html][Cython]]. Though in many cases 2881the benefits may not be obvious since the most computationally 2882intensive work never leaves the level of the C code with which GPGME 2883itself is interacting with. 2884 2885Nevertheless, there are some situations where the benefits are 2886demonstrable. One of the better and easier examples being the one of 2887the early examples in this HOWTO, the [[#howto-keys-counting][key counting]] code. Running that 2888example as an executable Python script, =keycount.py= (available in 2889the =examples/howto/= directory), will take a noticeable amount of time 2890to run on most systems where the public keybox or keyring contains a 2891few thousand public keys. 2892 2893Earlier in the evening, prior to starting this section, I ran that 2894script on my laptop; as I tend to do periodically and timed it using 2895=time= utility, with the following results: 2896 2897#+BEGIN_SRC shell 2898 bash-4.4$ time keycount.py 2899 2900 Number of secret keys: 23 2901 Number of public keys: 12112 2902 2903 2904 real 11m52.945s 2905 user 0m0.913s 2906 sys 0m0.752s 2907 2908 bash-4.4$ 2909#+END_SRC 2910 2911Sometime after that I imported another key and followed it with a 2912little test of Cython. This test was kept fairly basic, essentially 2913lifting the material from the [[http://docs.cython.org/en/latest/src/tutorial/cython_tutorial.html][Cython Basic Tutorial]] to demonstrate 2914compiling Python code to C. The first step was to take the example 2915key counting code quoted previously, essentially from the importing of 2916the =gpg= module to the end of the script: 2917 2918#+BEGIN_SRC python -i 2919import gpg 2920 2921c = gpg.Context() 2922seckeys = c.keylist(pattern=None, secret=True) 2923pubkeys = c.keylist(pattern=None, secret=False) 2924 2925seclist = list(seckeys) 2926secnum = len(seclist) 2927 2928publist = list(pubkeys) 2929pubnum = len(publist) 2930 2931print(""" 2932 Number of secret keys: {0} 2933 Number of public keys: {1} 2934 2935""".format(secnum, pubnum)) 2936#+END_SRC 2937 2938Save that into a file called =keycount.pyx= and then create a 2939=setup.py= file which contains this: 2940 2941#+BEGIN_SRC python -i 2942from distutils.core import setup 2943from Cython.Build import cythonize 2944 2945setup( 2946 ext_modules = cythonize("keycount.pyx") 2947) 2948#+END_SRC 2949 2950Compile it: 2951 2952#+BEGIN_SRC shell 2953 bash-4.4$ python setup.py build_ext --inplace 2954 bash-4.4$ 2955#+END_SRC 2956 2957Then run it in a similar manner to =keycount.py=: 2958 2959#+BEGIN_SRC shell 2960 bash-4.4$ time python3.7 -c "import keycount" 2961 2962 Number of secret keys: 23 2963 Number of public keys: 12113 2964 2965 2966 real 6m47.905s 2967 user 0m0.785s 2968 sys 0m0.331s 2969 2970 bash-4.4$ 2971#+END_SRC 2972 2973Cython turned =keycount.pyx= into an 81KB =keycount.o= file in the 2974=build/= directory, a 24KB =keycount.cpython-37m-darwin.so= file to be 2975imported into Python 3.7 and a 113KB =keycount.c= generated C source 2976code file of nearly three thousand lines. Quite a bit bigger than the 2977314 bytes of the =keycount.pyx= file or the full 1,452 bytes of the 2978full executable =keycount.py= example script. 2979 2980On the other hand it ran in nearly half the time; taking 6 minutes and 298147.905 seconds to run. As opposed to the 11 minutes and 52.945 seconds 2982which the CPython script alone took. 2983 2984The =keycount.pyx= and =setup.py= files used to generate this example 2985have been added to the =examples/howto/advanced/cython/= directory 2986The example versions include some additional options to annotate the 2987existing code and to detect Cython's use. The latter comes from the 2988[[http://docs.cython.org/en/latest/src/tutorial/pure.html#magic-attributes-within-the-pxd][Magic Attributes]] section of the Cython documentation. 2989 2990 2991* Miscellaneous extras and work-arounds 2992 :PROPERTIES: 2993 :CUSTOM_ID: cheats-and-hacks 2994 :END: 2995 2996Most of the things in the following sections are here simply because 2997there was no better place to put them, even though some are only 2998peripherally related to the GPGME Python bindings. Some are also 2999workarounds for functions not integrated with GPGME as yet. This is 3000especially true of the first of these, dealing with [[#group-lines][group lines]]. 3001 3002 3003** Group lines 3004 :PROPERTIES: 3005 :CUSTOM_ID: group-lines 3006 :END: 3007 3008There is not yet an easy way to access groups configured in the 3009gpg.conf file from within GPGME. As a consequence these central 3010groupings of keys cannot be shared amongst multiple programs, such as 3011MUAs readily. 3012 3013The following code, however, provides a work-around for obtaining this 3014information in Python. 3015 3016#+BEGIN_SRC python -i 3017import subprocess 3018import sys 3019 3020if sys.platform == "win32": 3021 gpgconfcmd = "gpgconf.exe --list-options gpg" 3022else: 3023 gpgconfcmd = "gpgconf --list-options gpg" 3024 3025process = subprocess.Popen(gpgconfcmd.split(), stdout=subprocess.PIPE) 3026procom = process.communicate() 3027 3028if sys.version_info[0] == 2: 3029 lines = procom[0].splitlines() 3030else: 3031 lines = procom[0].decode().splitlines() 3032 3033for line in lines: 3034 if line.startswith("group") is True: 3035 break 3036 3037groups = line.split(":")[-1].replace('"', '').split(',') 3038 3039group_lines = [] 3040group_lists = [] 3041 3042for group in groups: 3043 group_lines.append(group.split("=")) 3044 group_lists.append(group.split("=")) 3045 3046for glist in group_lists: 3047 glist[1] = glist[1].split() 3048#+END_SRC 3049 3050The result of that code is that =group_lines= is a list of lists where 3051=group_lines[i][0]= is the name of the group and =group_lines[i][1]= 3052is the key IDs of the group as a string. 3053 3054The =group_lists= result is very similar in that it is a list of 3055lists. The first part, =group_lists[i][0]= matches 3056=group_lines[i][0]= as the name of the group, but =group_lists[i][1]= 3057is the key IDs of the group as a list. 3058 3059A demonstration of using the =groups.py= module is also available in 3060the form of the executable =mutt-groups.py= script. This second 3061script reads all the group entries in a user's =gpg.conf= file and 3062converts them into crypt-hooks suitable for use with the Mutt and 3063Neomutt mail clients. 3064 3065 3066** Keyserver access for Python 3067 :PROPERTIES: 3068 :CUSTOM_ID: hkp4py 3069 :END: 3070 3071The [[https://github.com/Selfnet/hkp4py][hkp4py]] module by Marcel Fest was originally a port of the old 3072[[https://github.com/dgladkov/python-hkp][python-hkp]] module from Python 2 to Python 3 and updated to use the 3073[[http://docs.python-requests.org/en/latest/index.html][requests]] module instead. It has since been modified to provide 3074support for Python 2.7 as well and is available via PyPI. 3075 3076Since it rewrites the =hkp= protocol prefix as =http= and =hkps= as 3077=https=, the module is able to be used even with servers which do not 3078support the full scope of keyserver functions.[fn:5] It also works quite 3079readily when incorporated into a [[#cython][Cython]] generated and compiled version 3080of any code. 3081 3082 3083*** Key import format 3084 :PROPERTIES: 3085 :CUSTOM_ID: hkp4py-strings 3086 :END: 3087 3088The hkp4py module returns key data via requests as string literals 3089(=r.text=) instead of byte literals (=r.content=). This means that 3090the retrurned key data must be encoded to UTF-8 when importing that 3091key material using a =gpg.Context().key_import()= method. 3092 3093For this reason an alternative method has been added to the =search= 3094function of =hkp4py.KeyServer()= which returns the key in the correct 3095format as expected by =key_import=. When importing using this module, 3096it is now possible to import with this: 3097 3098#+BEGIN_SRC python -i 3099for key in keys: 3100 if key.revoked is False: 3101 gpg.Context().key_import(key.key_blob) 3102 else: 3103 pass 3104#+END_SRC 3105 3106Without that recent addition it would have been necessary to encode 3107the contents of each =hkp4py.KeyServer().search()[i].key= in 3108=hkp4py.KeyServer().search()= before trying to import it. 3109 3110An example of this is included in the [[#howto-import-key][Importing Keys]] section of this 3111HOWTO and the corresponding executable version of that example is 3112available in the =lang/python/examples/howto= directory as normal; the 3113executable version is the =import-keys-hkp.py= file. 3114 3115 3116** GPGME version checking 3117 :PROPERTIES: 3118 :CUSTOM_ID: gpgme-version-check 3119 :END: 3120 3121For various reasons it may be necessary to check which version of 3122GPGME the bindings have been built against; including whether a 3123minimum required version of GPGME is in use. 3124 3125For the most part the =gpg.version.versionstr= and 3126=gpg.version.versionlist= methods have been quite sufficient. The 3127former returns the same string as =gpgme-config --version=, while the 3128latter returns the major, minor and patch values in a list. 3129 3130To check if the installed bindings have actually been built against 3131the current installed libgpgme version, this check can be performed: 3132 3133#+BEGIN_SRC python -i 3134import gpg 3135import subprocess 3136import sys 3137 3138gpgme_version_call = subprocess.Popen(["gpgme-config", "--version"], 3139 stdout=subprocess.PIPE, 3140 stderr=subprocess.PIPE) 3141gpgme_version_str = gpgme_version_call.communicate() 3142 3143if sys.version_info[0] == 2: 3144 gpgme_version = gpgme_version_str[0].strip() 3145elif sys.version_info[0] >= 3: 3146 gpgme_version = gpgme_version_str[0].decode().strip() 3147else: 3148 gpgme_version = None 3149 3150if gpgme_version is not None: 3151 if gpgme_version == gpg.version.versionstr: 3152 print("The GPGME Python bindings match libgpgme.") 3153 else: 3154 print("The GPGME Python bindings do NOT match libgpgme.") 3155else: 3156 print("Upgrade Python and reinstall the GPGME Python bindings.") 3157#+END_SRC 3158 3159For many developers, however, the preferred checking means checking 3160for a minimum version or point release. This is now readily available 3161via the =gpg.version.versionintlist= method (added in version 3162=1.12.1-beta79=). It is also now possible to easily check whether the 3163installed GPGME Python bindings were built from a development or beta 3164branch of the GPGME source code. 3165 3166The following code demonstrates how both of those methods may be used: 3167 3168#+BEGIN_SRC python -i 3169import gpg 3170 3171try: 3172 if gpg.version.is_beta is True: 3173 print("The installed GPGME Python bindings were built from beta code.") 3174 else: 3175 print("The installed GPGME Python bindings are a released version.") 3176except Exception as e: 3177 print(e) 3178 3179try: 3180 if gpg.version.versionintlist[0] == 1: 3181 if gpg.version.versionintlist[1] == 12: 3182 if gpg.version.versionintlist[2] == 1: 3183 print("This is the minimum version for using versionintlist.") 3184 elif gpg.version.versionintlist[2] > 1: 3185 print("The versionintlist method is available.") 3186 else: 3187 pass 3188 elif gpg.version.versionintlist[1] > 12: 3189 print("The versionintlist method is available.") 3190 else: 3191 pass 3192 elif gpg.version.versionintlist[0] > 1: 3193 print("The versionintlist method is available.") 3194 else: 3195 pass 3196except Exception as e: 3197 print(e) 3198#+END_SRC 3199 3200The points where =pass= is used in the above example will most likely 3201also produce an =Exception= error since those results should only 3202occur in versions which do not have the =gpgme.version.is_beta= and 3203=gpgme.version.versionintlist= methods available. 3204 3205 3206* Copyright and Licensing 3207 :PROPERTIES: 3208 :CUSTOM_ID: copyright-and-license 3209 :END: 3210 3211 3212** Copyright 3213 :PROPERTIES: 3214 :CUSTOM_ID: copyright 3215 :END: 3216 3217Copyright © The GnuPG Project, 2018. 3218 3219 3220** Draft Editions of this HOWTO 3221 :PROPERTIES: 3222 :CUSTOM_ID: draft-editions 3223 :END: 3224 3225Draft editions of this HOWTO may be periodically available directly 3226from the author at any of the following URLs: 3227 3228- [[https://files.au.adversary.org/crypto/gpgme-python-howto.html][GPGME Python Bindings HOWTO draft (HTML single file, AWS S3 SSL)]] 3229- [[http://files.au.adversary.org/crypto/gpgme-python-howto.html][GPGME Python Bindings HOWTO draft (HTML single file, AWS S3 no SSL)]] 3230- [[https://files.au.adversary.org/crypto/gpgme-python-howto-split/index.html][GPGME Python Bindings HOWTO draft (HTML multiple files, AWS S3 SSL)]] 3231- [[http://files.au.adversary.org/crypto/gpgme-python-howto/index.html][GPGME Python Bindings HOWTO draft (HTML multiple files, AWS S3 no SSL)]] 3232 3233These draft versions have been generated from this document via GNU 3234Emacs [[https://orgmode.org/][Org mode]] to =.texi= and [[https://www.gnu.org/software/texinfo/][GNU Texinfo]] to HTML. Though it is 3235likely that the specific [[https://files.au.adversary.org/crypto/gpgme-python-howto][file]] [[http://files.au.adversary.org/crypto/gpgme-python-howto.org][version]] used will be on the same server 3236with the generated output formats. Occasionally I may include the Org 3237mode generated XHTML versions: 3238 3239- [[https://files.au.adversary.org/crypto/gpgme-python-howto.xhtml][GPGME Python Bindings HOWTO draft (HTML single file, AWS S3 SSL)]] 3240- [[http://files.au.adversary.org/crypto/gpgme-python-howto.xhtml][GPGME Python Bindings HOWTO draft (HTML single file, AWS S3 no SSL)]] 3241 3242That XHTML version, however, is exported in a way which inherits a 3243colour scheme from [[https://github.com/holomorph/emacs-zenburn][the author's Emacs theme]] (which is a higher contrast 3244version of [[http://kippura.org/zenburnpage/][Zenburn]] ported by [[https://github.com/holomorph][Holomorph]]). So it's fine for people who 3245prefer dark themed web pages, but not so great for everyone else. 3246 3247The GNU Texinfo and reStructured Text versions ship with the software, 3248while the GNU Emacs Info version is generated from the Texinfo 3249version using GNU Texinfo or GNU Makeinfo. The Texinfo format is 3250generated from the original Org mode source file in Org mode itself 3251either within GNU Emacs or via the command line by invoking Emacs in 3252batch mode: 3253 3254#+BEGIN_SRC shell 3255 emacs gpgme-python-howto.org --batch -f org-texinfo-export-to-texinfo --kill 3256 emacs gpgme-python-howto --batch -f org-texinfo-export-to-texinfo --kill 3257#+END_SRC 3258 3259The reStructuredText format is also generated from the Org mode source 3260file, except it is generated using [[https://pandoc.org][Pandoc]] with either of the following 3261commands (depending on the filename): 3262 3263#+BEGIN_SRC shell 3264 pandoc -f org -t rst+smart -o gpgme-python-howto.rst gpgme-python-howto.org 3265 pandoc -f org -t rst+smart -o gpgme-python-howto.rst gpgme-python-howto 3266#+END_SRC 3267 3268Note that the Org mode source files are identified as such via a mode 3269line at the top of each file and have had their =.org= file extensions 3270dropped in order to make scripted generation of output formats easier 3271and not require renaming files post-conversion. 3272 3273Due to a bug in Org mode's texinfo conversion method, the recommended 3274steps for generating the Texinfo files for all the files in the 3275=lang/python/doc/src/= directory are as follows: 3276 3277#+BEGIN_SRC shell 3278 for x in * ; do 3279 emacs $x --batch -f org-texinfo-export-to-texinfo --kill 3280 cat $x.texi | sed -e 's/@documentencoding UTF-8/@documentencoding utf-8/g' > ../texinfo/$x.texi 3281 pandoc -f org -t rst+smart -o ../rst/$x.rst $x 3282 done ; 3283 rm -fv *.texi 3284 cd ../texinfo 3285 mkdir info 3286 mkdir html 3287 for x in *.texi ; do 3288 makeinfo -v $x 3289 makeinfo --html --no-split $x 3290 done ; 3291 mv *.info info/ 3292 mv *.html html/ 3293#+END_SRC 3294 3295This code snippet includes the generation of the reStructuredText 3296files and would be expected to be run from the =doc/src/= directory 3297containing the Org mode source files. It also assumes that the 3298commands are being run on POSIX compliant systems with basic tools 3299like sed, the Bourne shell and GNU Emacs[fn:6] available. The code 3300snippet also includes the steps for generating the Emacs Info files 3301and HTML files from the Texinfo files. Using reStructuredText files 3302with Sphinx is best left for the documentation of that project. 3303 3304In addition to these there is a significantly less frequently updated 3305version as a HTML [[https://files.au.adversary.org/crypto/gpgme-python/dita/webhelp/index.html][WebHelp site]] (AWS S3 SSL); generated from DITA XML 3306source files, which can be found in [[https://dev.gnupg.org/source/gpgme/browse/ben%252Fhowto-dita/][an alternative branch]] of the GPGME 3307git repository. 3308 3309Various generated output formats may occasionally be found in 3310subdirectories of the [[https://s3.amazonaws.com/files.au.adversary.org/crypto/gpgme-python][gpgme-python]] directory. In particular within 3311the [[https://s3.amazonaws.com/files.au.adversary.org/crypto/gpgme-python/dita][DITA]], [[https://s3.amazonaws.com/files.au.adversary.org/crypto/gpgme-python/rst][reStructuredText]] and [[https://s3.amazonaws.com/files.au.adversary.org/crypto/gpgme-python/texinfo][Texinfo]] subdirectories. The =rst= 3312directory contains output files generated with Sphinx and may include a 3313considerable number of its possible output formats, but there are no 3314guarantees as to how recent these are or even if they are present. 3315 3316These draft editions are not official documents and the version of 3317documentation in the master branch or which ships with released 3318versions is the only official documentation. Nevertheless, these 3319draft editions may occasionally be of use by providing more accessible 3320web versions which are updated between releases. They are provided on 3321the understanding that they may contain errors or may contain content 3322subject to change prior to an official release. 3323 3324 3325** License GPL compatible 3326 :PROPERTIES: 3327 :CUSTOM_ID: license 3328 :END: 3329 3330This file is free software; as a special exception the author gives 3331unlimited permission to copy and/or distribute it, with or without 3332modifications, as long as this notice is preserved. 3333 3334This file is distributed in the hope that it will be useful, but 3335WITHOUT ANY WARRANTY, to the extent permitted by law; without even the 3336implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR 3337PURPOSE. 3338 3339 3340* Footnotes 3341 3342[fn:1] =short-history= and/or =short-history.html=. 3343 3344[fn:2] With no issues reported specific to Python 3.7, the release of 3345Python 3.7.1 at around the same time as GPGME 1.12.0 and the testing 3346with Python 3.7.1rc1, there is no reason to delay moving 3.7 ahead of 33473.6 now. Production environments with more conservative requirements 3348will always enforce their own policies anyway and installation to each 3349supported minor release is quite possible too. 3350 3351[fn:3] Yes, even if you use virtualenv with everything you do in 3352Python. If you want to install this module as just your user account 3353then you will need to manually configure, compile and install the 3354/entire/ GnuPG stack as that user as well. This includes libraries 3355which are not often installed that way. It can be done and there are 3356circumstances under which it is worthwhile, but generally only on 3357POSIX systems which utilise single user mode (some even require it). 3358 3359[fn:4] You probably don't really want to do this. Searching the 3360keyservers for "gnupg.org" produces over 400 results, the majority of 3361which aren't actually at the gnupg.org domain, but just included a 3362comment regarding the project in their key somewhere. 3363 3364[fn:5] Such as with ProtonMail servers. This also means that 3365restricted servers which only advertise either HTTP or HTTPS end 3366points and not HKP or HKPS end points must still be identified as as 3367HKP or HKPS within the Python Code. The =hkp4py= module will rewrite 3368these appropriately when the connection is made to the server. 3369 3370[fn:6] Okay, Emacs might not necessarily qualify as a basic tool, but 3371it is common enough that having it installed on a system isn't too 3372great an expectation, nor is it difficult to add to most POSIX 3373systems, even if the users of those systems do not personally use it. 3374