diff options
author | Ben McGinnes <[email protected]> | 2018-10-04 11:10:46 +0000 |
---|---|---|
committer | Ben McGinnes <[email protected]> | 2018-10-04 11:10:46 +0000 |
commit | f3fc73738264d21baf0e9b49fdd5cb67faa11d31 (patch) | |
tree | 339b96e365b5cf849443431f743aa983671db022 /lang/python/doc/rst/gpgme-python-howto.rst | |
parent | python: make file (diff) | |
download | gpgme-f3fc73738264d21baf0e9b49fdd5cb67faa11d31.tar.gz gpgme-f3fc73738264d21baf0e9b49fdd5cb67faa11d31.zip |
python: makefiles
* doc/Makefile.am: Removed a bit I forgot about.
* Renamed lang/python/docs to lang/python/doc bvecause apparently
automake cares about that too.
* Decided to be extra explicit in the manifest because if I don't then
all sorts of things get deleted ... like lang/python
* Tested on an external linux system just in case my osx workstation
introduces too much weirdness.
Tested-by: Ben McGinnes <[email protected]>
Signed-off-by: Ben McGinnes <[email protected]>
Diffstat (limited to 'lang/python/doc/rst/gpgme-python-howto.rst')
-rw-r--r-- | lang/python/doc/rst/gpgme-python-howto.rst | 3001 |
1 files changed, 3001 insertions, 0 deletions
diff --git a/lang/python/doc/rst/gpgme-python-howto.rst b/lang/python/doc/rst/gpgme-python-howto.rst new file mode 100644 index 00000000..f29727e6 --- /dev/null +++ b/lang/python/doc/rst/gpgme-python-howto.rst @@ -0,0 +1,3001 @@ +.. _intro: + +Introduction +============ + ++-----------------------------------+-----------------------------------+ +| Version: | 0.1.4 | ++-----------------------------------+-----------------------------------+ +| GPGME Version: | 1.12.0 | ++-----------------------------------+-----------------------------------+ +| Author: | `Ben | +| | McGinnes <https://gnupg.org/peopl | +| | e/index.html#sec-1-5>`__ | +| | <[email protected]> | ++-----------------------------------+-----------------------------------+ +| Author GPG Key: | DB4724E6FA4286C92B4E55C4321E4E237 | +| | 3590E5D | ++-----------------------------------+-----------------------------------+ +| Language: | Australian English, British | +| | English | ++-----------------------------------+-----------------------------------+ +| xml:lang: | en-AU, en-GB, en | ++-----------------------------------+-----------------------------------+ + +This document provides basic instruction in how to use the GPGME Python +bindings to programmatically leverage the GPGME library. + +.. _py2-vs-py3: + +Python 2 versus Python 3 +------------------------ + +Though the GPGME Python bindings themselves provide support for both +Python 2 and 3, the focus is unequivocally on Python 3 and specifically +from Python 3.4 and above. As a consequence all the examples and +instructions in this guide use Python 3 code. + +Much of it will work with Python 2, but much of it also deals with +Python 3 byte literals, particularly when reading and writing data. +Developers concentrating on Python 2.7, and possibly even 2.6, will need +to make the appropriate modifications to support the older string and +unicode types as opposed to bytes. + +There are multiple reasons for concentrating on Python 3; some of which +relate to the immediate integration of these bindings, some of which +relate to longer term plans for both GPGME and the python bindings and +some of which relate to the impending EOL period for Python 2.7. +Essentially, though, there is little value in tying the bindings to a +version of the language which is a dead end and the advantages offered +by Python 3 over Python 2 make handling the data types with which GPGME +deals considerably easier. + +.. _howto-python3-examples: + +Examples +-------- + +All of the examples found in this document can be found as Python 3 +scripts in the ``lang/python/examples/howto`` directory. + +Unofficial Drafts +----------------- + +In addition to shipping with each release of GPGME, there is a section +on locations to read or download `draft editions <#draft-editions>`__ of +this document from at the end of it. These are unofficial versions +produced in between major releases. + +.. _new-stuff: + +What\'s New +----------- + +The most obviously new point for those reading this guide is this +section on other new things, but that\'s hardly important. Not given all +the other things which spurred the need for adding this section and its +subsections. + +.. _new-stuff-1-12-0: + +New in GPGME 1·12·0 +~~~~~~~~~~~~~~~~~~~ + +There have been quite a number of additions to GPGME and the Python +bindings to it since the last release of GPGME with versions 1.11.0 and +1.11.1 in April, 2018. + +The bullet points of new additiions are: + +- an expanded section on `installing <#installation>`__ and + `troubleshooting <#snafu>`__ the Python bindings. +- The release of Python 3.7.0; which appears to be working just fine + with our bindings, in spite of intermittent reports of problems for + many other Python projects with that new release. +- Python 3.7 has been moved to the head of the specified python + versions list in the build process. +- In order to fix some other issues, there are certain underlying + functions which are more exposed through the + `gpg.Context() <#howto-get-context>`__, but ongoing documentation + ought to clarify that or otherwise provide the best means of using + the bindings. Some additions to ``gpg.core`` and the ``Context()``, + however, were intended (see below). +- Continuing work in identifying and confirming the cause of + oft-reported `problems installing the Python bindings on + Windows <#snafu-runtime-not-funtime>`__. +- GSOC: Google\'s Surreptitiously Ordered Conscription ... erm ... oh, + right; Google\'s Summer of Code. Though there were two hopeful + candidates this year; only one ended up involved with the GnuPG + Project directly, the other concentrated on an unrelated third party + project with closer ties to one of the GNU/Linux distributions than + to the GnuPG Project. Thus the Python bindings benefited from GSOC + participant Jacob Adams, who added the key\ :sub:`import` function; + building on prior work by Tobias Mueller. +- Several new methods functions were added to the gpg.Context(), + including: `key\ import <#howto-import-key>`__, + `key\ export <#howto-export-key>`__, + `key\ exportminimal <#howto-export-public-key>`__ and + `key\ exportsecret <#howto-export-secret-key>`__. +- Importing and exporting examples include versions integrated with + Marcel Fest\'s recently released `HKP for + Python <https://github.com/Selfnet/hkp4py>`__ module. Some + `additional notes on this module <#hkp4py>`__ are included at the end + of the HOWTO. +- Instructions for dealing with semi-walled garden implementations like + ProtonMail are also included. This is intended to make things a + little easier when communicating with users of ProtonMail\'s services + and should not be construed as an endorsement of said service. The + GnuPG Project neither favours, nor disfavours ProtonMail and the + majority of this deals with interacting with the ProtonMail + keyserver. +- Semi-formalised the location where `draft + versions <#draft-editions>`__ of this HOWTO may periodically be + accessible. This is both for the reference of others and testing the + publishing of the document itself. Renamed this file at around the + same time. +- The Texinfo documentation build configuration has been replicated + from the parent project in order to make to maintain consistency with + that project (and actually ship with each release). +- a reStructuredText (``.rst``) version is also generated for Python + developers more used to and comfortable with that format as it is the + standard Python documentation format and Python developers may wish + to use it with Sphinx. Please note that there has been no testing of + the reStructuredText version with Sphinx at all. The reST file was + generated by the simple expedient of using + `Pandoc <https://pandoc.org/>`__. +- Added a new section for `advanced or experimental + use <#advanced-use>`__. +- Began the advanced use cases with `a section <#cython>`__ on using + the module with `Cython <http://cython.org/>`__. +- Added a number of new scripts to the ``example/howto/`` directory; + some of which may be in advance of their planned sections of the + HOWTO (and some are just there because it seemed like a good idea at + the time). +- Cleaned up a lot of things under the hood. + +GPGME Concepts +============== + +.. _gpgme-c-api: + +A C API +------- + +Unlike many modern APIs with which programmers will be more familiar +with these days, the GPGME API is a C API. The API is intended for use +by C coders who would be able to access its features by including the +``gpgme.h`` header file with their own C source code and then access its +functions just as they would any other C headers. + +This is a very effective method of gaining complete access to the API +and in the most efficient manner possible. It does, however, have the +drawback that it cannot be directly used by other languages without some +means of providing an interface to those languages. This is where the +need for bindings in various languages stems. + +.. _gpgme-python-bindings: + +Python bindings +--------------- + +The Python bindings for GPGME provide a higher level means of accessing +the complete feature set of GPGME itself. It also provides a more +pythonic means of calling these API functions. + +The bindings are generated dynamically with SWIG and the copy of +``gpgme.h`` generated when GPGME is compiled. + +This means that a version of the Python bindings is fundamentally tied +to the exact same version of GPGME used to generate that copy of +``gpgme.h``. + +.. _gpgme-python-bindings-diffs: + +Difference between the Python bindings and other GnuPG Python packages +---------------------------------------------------------------------- + +There have been numerous attempts to add GnuPG support to Python over +the years. Some of the most well known are listed here, along with what +differentiates them. + +.. _diffs-python-gnupg: + +The python-gnupg package maintained by Vinay Sajip +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is arguably the most popular means of integrating GPG with Python. +The package utilises the ``subprocess`` module to implement wrappers for +the ``gpg`` and ``gpg2`` executables normally invoked on the command +line (``gpg.exe`` and ``gpg2.exe`` on Windows). + +The popularity of this package stemmed from its ease of use and +capability in providing the most commonly required features. + +Unfortunately it has been beset by a number of security issues in the +past; most of which stemmed from using unsafe methods of accessing the +command line via the ``subprocess`` calls. While some effort has been +made over the last two to three years (as of 2018) to mitigate this, +particularly by no longer providing shell access through those +subprocess calls, the wrapper is still somewhat limited in the scope of +its GnuPG features coverage. + +The python-gnupg package is available under the MIT license. + +.. _diffs-isis-gnupg: + +The gnupg package created and maintained by Isis Lovecruft +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In 2015 Isis Lovecruft from the Tor Project forked and then +re-implemented the python-gnupg package as just gnupg. This new package +also relied on subprocess to call the ``gpg`` or ``gpg2`` binaries, but +did so somewhat more securely. + +The naming and version numbering selected for this package, however, +resulted in conflicts with the original python-gnupg and since its +functions were called in a different manner to python-gnupg, the release +of this package also resulted in a great deal of consternation when +people installed what they thought was an upgrade that subsequently +broke the code relying on it. + +The gnupg package is available under the GNU General Public License +version 3.0 (or any later version). + +.. _diffs-pyme: + +The PyME package maintained by Martin Albrecht +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This package is the origin of these bindings, though they are somewhat +different now. For details of when and how the PyME package was folded +back into GPGME itself see the *Short History* document [1]_ in the +Python bindings ``docs`` directory. [2]_ + +The PyME package was first released in 2002 and was also the first +attempt to implement a low level binding to GPGME. In doing so it +provided access to considerably more functionality than either the +``python-gnupg`` or ``gnupg`` packages. + +The PyME package is only available for Python 2.6 and 2.7. + +Porting the PyME package to Python 3.4 in 2015 is what resulted in it +being folded into the GPGME project and the current bindings are the end +result of that effort. + +The PyME package is available under the same dual licensing as GPGME +itself: the GNU General Public License version 2.0 (or any later +version) and the GNU Lesser General Public License version 2.1 (or any +later version). + +.. _gpgme-python-install: + +GPGME Python bindings installation +================================== + +.. _do-not-use-pypi: + +No PyPI +------- + +Most third-party Python packages and modules are available and +distributed through the Python Package Installer, known as PyPI. + +Due to the nature of what these bindings are and how they work, it is +infeasible to install the GPGME Python bindings in the same way. + +This is because the bindings use SWIG to dynamically generate C bindings +against ``gpgme.h`` and ``gpgme.h`` is generated from ``gpgme.h.in`` at +compile time when GPGME is built from source. Thus to include a package +in PyPI which actually built correctly would require either statically +built libraries for every architecture bundled with it or a full +implementation of C for each architecture. + +See the additional notes regarding `CFFI and SWIG <#snafu-cffi>`__ at +the end of this section for further details. + +.. _gpgme-python-requirements: + +Requirements +------------ + +The GPGME Python bindings only have three requirements: + +#. A suitable version of Python 2 or Python 3. With Python 2 that means + CPython 2.7 and with Python 3 that means CPython 3.4 or higher. +#. `SWIG <https://www.swig.org>`__. +#. GPGME itself. Which also means that all of GPGME\'s dependencies must + be installed too. + +.. _gpgme-python-recommendations: + +Recommended Additions +~~~~~~~~~~~~~~~~~~~~~ + +Though none of the following are absolute requirements, they are all +recommended for use with the Python bindings. In some cases these +recommendations refer to which version(s) of CPython to use the bindings +with, while others refer to third party modules which provide a +significant advantage in some way. + +#. If possible, use Python 3 instead of 2. +#. Favour a more recent version of Python since even 3.4 is due to reach + EOL soon. In production systems and services, Python 3.6 should be + robust enough to be relied on. +#. If possible add the following Python modules which are not part of + the standard library: + `Requests <http://docs.python-requests.org/en/latest/index.html>`__, + `Cython <http://cython.org/>`__ and + `hkp4py <https://github.com/Selfnet/hkp4py>`__. Chances are quite + high that at least the first one and maybe two of those will already + be installed. + +Note that, as with Cython, some of the planned additions to the +`Advanced <#advanced-use>`__ section, will bring with them additional +requirements. Most of these will be fairly well known and commonly +installed ones, however, which are in many cases likely to have already +been installed on many systems or be familiar to Python programmers. + +Installation +------------ + +Installing the Python bindings is effectively achieved by compiling and +installing GPGME itself. + +Once SWIG is installed with Python and all the dependencies for GPGME +are installed you only need to confirm that the version(s) of Python you +want the bindings installed for are in your ``$PATH``. + +By default GPGME will attempt to install the bindings for the most +recent or highest version number of Python 2 and Python 3 it detects in +``$PATH``. It specifically checks for the ``python`` and ``python3`` +executables first and then checks for specific version numbers. + +For Python 2 it checks for these executables in this order: ``python``, +``python2`` and ``python2.7``. + +For Python 3 it checks for these executables in this order: ``python3``, +``python3.7``, ``python3.6``, ``python3.5`` and ``python3.4``. [3]_ + +On systems where ``python`` is actually ``python3`` and not ``python2`` +it may be possible that ``python2`` may be overlooked, but there have +been no reports of that actually occurring as yet. + +In the three months or so since the release of Python 3.7.0 there has +been extensive testing and work with these bindings with no issues +specifically relating to the new version of Python or any of the new +features of either the language or the bindings. This has also been the +case with Python 3.7.1rc1. With that in mind and given the release of +Python 3.7.1 is scheduled for around the same time as GPGME 1.12.0, the +order of preferred Python versions has been changed to move Python 3.7 +ahead of Python 3.6. + +.. _install-gpgme: + +Installing GPGME +~~~~~~~~~~~~~~~~ + +See the GPGME ``README`` file for details of how to install GPGME from +source. + +.. _snafu: + +Known Issues +------------ + +There are a few known issues with the current build process and the +Python bindings. For the most part these are easily addressed should +they be encountered. + +.. _snafu-a-swig-of-this-builds-character: + +Breaking Builds +~~~~~~~~~~~~~~~ + +Occasionally when installing GPGME with the Python bindings included it +may be observed that the ``make`` portion of that process induces a +large very number of warnings and, eventually errors which end that part +of the build process. Yet following that with ``make check`` and +``make install`` appears to work seamlessly. + +The cause of this is related to the way SWIG needs to be called to +dynamically generate the C bindings for GPGME in the first place. So the +entire process will always produce ``lang/python/python2-gpg/`` and +``lang/python/python3-gpg/`` directories. These should contain the build +output generated during compilation, including the complete bindings and +module installed into ``site-packages``. + +Occasionally the errors in the early part or some other conflict (e.g. +not installing as **root** or **su**) may result in nothing being +installed to the relevant ``site-packages`` directory and the build +directory missing a lot of expected files. Even when this occurs, the +solution is actually quite simple and will always work. + +That solution is simply to run the following commands as either the +**root** user or prepended with ``sudo -H``\ [4]_ in the +``lang/python/`` directory: + +.. code:: shell + + /path/to/pythonX.Y setup.py build + /path/to/pythonX.Y setup.py build + /path/to/pythonX.Y setup.py install + +Yes, the build command does need to be run twice. Yes, you still need to +run the potentially failing or incomplete steps during the +``configure``, ``make`` and ``make install`` steps with installing +GPGME. This is because those steps generate a lot of essential files +needed, both by and in order to create, the bindings (including both the +``setup.py`` and ``gpgme.h`` files). + +#. IMPORTANT Note + + If specifying a selected number of languages to create bindings for, + try to leave Python last. Currently the majority of the other + language bindings are also preceding Python of either version when + listed alphabetically and so that just happens by default currently. + + If Python is set to precede one of the other languages then it is + possible that the errors described here may interrupt the build + process before generating bindings for those other languages. In + these cases it may be preferable to configure all preferred language + bindings separately with alternative ``configure`` steps for GPGME + using the ``--enable-languages=$LANGUAGE`` option. + +.. _snafu-lessons-for-the-lazy: + +Reinstalling Responsibly +~~~~~~~~~~~~~~~~~~~~~~~~ + +Regardless of whether you\'re installing for one version of Python or +several, there will come a point where reinstallation is required. With +most Python module installations, the installed files go into the +relevant site-packages directory and are then forgotten about. Then the +module is upgraded, the new files are copied over the old and that\'s +the end of the matter. + +While the same is true of these bindings, there have been intermittent +issues observed on some platforms which have benefited significantly +from removing all the previous installations of the bindings before +installing the updated versions. + +Removing the previous version(s) is simply a matter of changing to the +relevant ``site-packages`` directory for the version of Python in +question and removing the ``gpg/`` directory and any accompanying +egg-info files for that module. + +In most cases this will require root or administration privileges on the +system, but the same is true of installing the module in the first +place. + +.. _snafu-the-full-monty: + +Multiple installations +~~~~~~~~~~~~~~~~~~~~~~ + +For a veriety of reasons it may be either necessary or just preferable +to install the bindings to alternative installed Python versions which +meet the requirements of these bindings. + +On POSIX systems this will generally be most simply achieved by running +the manual installation commands (build, build, install) as described in +the previous section for each Python installation the bindings need to +be installed to. + +As per the SWIG documentation: the compilers, libraries and runtime used +to build GPGME and the Python Bindings **must** match those used to +compile Python itself, including the version number(s) (at least going +by major version numbers and probably minor numbers too). + +On most POSIX systems, including OS X, this will very likely be the case +in most, if not all, cases. + +.. _snafu-runtime-not-funtime: + +Won\'t Work With Windows +~~~~~~~~~~~~~~~~~~~~~~~~ + +There are semi-regular reports of Windows users having considerable +difficulty in installing and using the Python bindings at all. Very +often, possibly even always, these reports come from Cygwin users and/or +MinGW users and/or Msys2 users. Though not all of them have been +confirmed, it appears that these reports have also come from people who +installed Python using the Windows installer files from the `Python +website <https://python.org>`__ (i.e. mostly MSI installers, sometimes +self-extracting ``.exe`` files). + +The Windows versions of Python are not built using Cygwin, MinGW or +Msys2; they\'re built using Microsoft Visual Studio. Furthermore the +version used is *considerably* more advanced than the version which +MinGW obtained a small number of files from many years ago in order to +be able to compile anything at all. Not only that, but there are changes +to the version of Visual Studio between some micro releases, though that +is is particularly the case with Python 2.7, since it has been kept +around far longer than it should have been. + +There are two theoretical solutions to this issue: + +#. Compile and install the GnuPG stack, including GPGME and the Python + bibdings using the same version of Microsoft Visual Studio used by + the Python Foundation to compile the version of Python installed. + + If there are multiple versions of Python then this will need to be + done with each different version of Visual Studio used. + +#. Compile and install Python using the same tools used by choice, such + as MinGW or Msys2. + +Do **not** use the official Windows installer for Python unless +following the first method. + +In this type of situation it may even be for the best to accept that +there are less limitations on permissive software than free software and +simply opt to use a recent version of the Community Edition of Microsoft +Visual Studio to compile and build all of it, no matter what. + +Investigations into the extent or the limitations of this issue are +ongoing. + +.. _snafu-cffi: + +CFFI is the Best™ and GPGME should use it instead of SWIG +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are many reasons for favouring +`CFFI <https://cffi.readthedocs.io/en/latest/overview.html>`__ and +proponents of it are quite happy to repeat these things as if all it +would take to switch from SWIG to CFFI is repeating that list as if it +were a new concept. + +The fact is that there are things which Python\'s CFFI implementation +cannot handle in the GPGME C code. Beyond that there are features of +SWIG which are simply not available with CFFI at all. SWIG generates the +bindings to Python using the ``gpgme.h`` file, but that file is not a +single version shipped with each release, it too is generated when GPGME +is compiled. + +CFFI is currently unable to adapt to such a potentially mutable +codebase. If there were some means of applying SWIG\'s dynamic code +generation to produce the Python/CFFI API modes of accessing the GPGME +libraries (or the source source code directly), but such a thing does +not exist yet either and it currently appears that work is needed in at +least one of CFFI\'s dependencies before any of this can be addressed. + +So if you\'re a massive fan of CFFI; that\'s great, but if you want this +project to switch to CFFI then rather than just insisting that it +should, I\'d suggest you volunteer to bring CFFI up to the level this +project needs. + +If you\'re actually seriously considering doing so, then I\'d suggest +taking the ``gpgme-tool.c`` file in the GPGME ``src/`` directory and +getting that to work with any of the CFFI API methods (not the ABI +methods, they\'ll work with pretty much anything). When you start +running into trouble with \"ifdefs\" then you\'ll know what sort of +things are lacking. That doesn\'t even take into account the amount of +work saved via SWIG\'s code generation techniques either. + +.. _snafu-venv: + +Virtualised Environments +~~~~~~~~~~~~~~~~~~~~~~~~ + +It is fairly common practice amongst Python developers to, as much as +possible, use packages like virtualenv to keep various things that are +to be installed from interfering with each other. Given how much of the +GPGME bindings is often at odds with the usual pythonic way of doing +things, it stands to reason that this would be called into question too. + +As it happens the answer as to whether or not the bindings can be used +with virtualenv, the answer is both yes and no. + +In general we recommend installing to the relevant path and matching +prefix of GPGME itself. Which means that when GPGME, and ideally the +rest of the GnuPG stack, is installed to a prefix like ``/usr/local`` or +``/opt/local`` then the bindings would need to be installed to the main +Python installation and not a virtualised abstraction. Attempts to +separate the two in the past have been known to cause weird and +intermittent errors ranging from minor annoyances to complete failures +in the build process. + +As a consequence we only recommend building with and installing to the +main Python installations within the same prefix as GPGME is installed +to or which are found by GPGME\'s configuration stage immediately prior +to running the make commands. Which is exactly what the compiling and +installing process of GPGME does by default. + +Once that is done, however, it appears that a copy the compiled module +may be installed into a virtualenv of the same major and minor version +matching the build. Alternatively it is possible to utilise a +``sites.pth`` file in the ``site-packages/`` directory of a viertualenv +installation, which links back to the system installations corresponding +directory in order to import anything installed system wide. This may or +may not be appropriate on a case by case basis. + +Though extensive testing of either of these options is not yet complete, +preliminary testing of them indicates that both are viable as long as +the main installation is complete. Which means that certain other +options normally restricted to virtual environments are also available, +including integration with pythonic test suites (e.g. +`pytest <https://docs.pytest.org/en/latest/index.html>`__) and other +large projects. + +That said, it is worth reiterating the warning regarding non-standard +installations. If one were to attempt to install the bindings only to a +virtual environment without somehow also including the full GnuPG stack +(or enough of it as to include GPGME) then it is highly likely that +errors would be encountered at some point and more than a little likely +that the build process itself would break. + +If a degree of separation from the main operating system is still +required in spite of these warnings, then consider other forms of +virtualisation. Either a virtual machine (e.g. +`VirtualBox <https://www.virtualbox.org/>`__), a hardware emulation +layer (e.g. `QEMU <https://www.qemu.org/>`__) or an application +container (e.g. `Docker <https://www.docker.com/why-docker>`__). + +Finally it should be noted that the limited tests conducted thus far +have been using the ``virtualenv`` command in a new directory to create +the virtual python environment. As opposed to the standard ``python3 +-m venv`` and it is possible that this will make a difference depending +on the system and version of Python in use. Another option is to run the +command ``python3 -m virtualenv /path/to/install/virtual/thingy`` +instead. + +.. _howto-fund-a-mental: + +Fundamentals +============ + +Before we can get to the fun stuff, there are a few matters regarding +GPGME\'s design which hold true whether you\'re dealing with the C code +directly or these Python bindings. + +.. _no-rest-for-the-wicked: + +No REST +------- + +The first part of which is or will be fairly blatantly obvious upon +viewing the first example, but it\'s worth reiterating anyway. That +being that this API is **not** a REST API. Nor indeed could it ever be +one. + +Most, if not all, Python programmers (and not just Python programmers) +know how easy it is to work with a RESTful API. In fact they\'ve become +so popular that many other APIs attempt to emulate REST-like behaviour +as much as they are able. Right down to the use of JSON formatted output +to facilitate the use of their API without having to retrain developers. + +This API does not do that. It would not be able to do that and also +provide access to the entire C API on which it\'s built. It does, +however, provide a very pythonic interface on top of the direct bindings +and it\'s this pythonic layer that this HOWTO deals with. + +.. _howto-get-context: + +Context +------- + +One of the reasons which prevents this API from being RESTful is that +most operations require more than one instruction to the API to perform +the task. Sure, there are certain functions which can be performed +simultaneously, particularly if the result known or strongly anticipated +(e.g. selecting and encrypting to a key known to be in the public +keybox). + +There are many more, however, which cannot be manipulated so readily: +they must be performed in a specific sequence and the result of one +operation has a direct bearing on the outcome of subsequent operations. +Not merely by generating an error either. + +When dealing with this type of persistent state on the web, full of both +the RESTful and REST-like, it\'s most commonly referred to as a session. +In GPGME, however, it is called a context and every operation type has +one. + +.. _howto-keys: + +Working with keys +================= + +.. _howto-keys-selection: + +Key selection +------------- + +Selecting keys to encrypt to or to sign with will be a common occurrence +when working with GPGMe and the means available for doing so are quite +simple. + +They do depend on utilising a Context; however once the data is recorded +in another variable, that Context does not need to be the same one which +subsequent operations are performed. + +The easiest way to select a specific key is by searching for that key\'s +key ID or fingerprint, preferably the full fingerprint without any +spaces in it. A long key ID will probably be okay, but is not advised +and short key IDs are already a problem with some being generated to +match specific patterns. It does not matter whether the pattern is upper +or lower case. + +So this is the best method: + +.. code:: python + + import gpg + + k = gpg.Context().keylist(pattern="258E88DCBD3CD44D8E7AB43F6ECB6AF0DEADBEEF") + keys = list(k) + +This is passable and very likely to be common: + +.. code:: python + + import gpg + + k = gpg.Context().keylist(pattern="0x6ECB6AF0DEADBEEF") + keys = list(k) + +And this is a really bad idea: + +.. code:: python + + import gpg + + k = gpg.Context().keylist(pattern="0xDEADBEEF") + keys = list(k) + +Alternatively it may be that the intention is to create a list of keys +which all match a particular search string. For instance all the +addresses at a particular domain, like this: + +.. code:: python + + import gpg + + ncsc = gpg.Context().keylist(pattern="ncsc.mil") + nsa = list(ncsc) + +.. _howto-keys-counting: + +Counting keys +~~~~~~~~~~~~~ + +Counting the number of keys in your public keybox (``pubring.kbx``), the +format which has superseded the old keyring format (``pubring.gpg`` and +``secring.gpg``), or the number of secret keys is a very simple task. + +.. code:: python + + import gpg + + c = gpg.Context() + seckeys = c.keylist(pattern=None, secret=True) + pubkeys = c.keylist(pattern=None, secret=False) + + seclist = list(seckeys) + secnum = len(seclist) + + publist = list(pubkeys) + pubnum = len(publist) + + print(""" + Number of secret keys: {0} + Number of public keys: {1} + """.format(secnum, pubnum)) + +NOTE: The `Cython <#cython>`__ introduction in the `Advanced and +Experimental <#advanced-use>`__ section uses this same key counting code +with Cython to demonstrate some areas where Cython can improve +performance even with the bindings. Users with large public keyrings or +keyboxes, for instance, should consider these options if they are +comfortable with using Cython. + +.. _howto-get-key: + +Get key +------- + +An alternative method of getting a single key via its fingerprint is +available directly within a Context with ``Context().get_key``. This is +the preferred method of selecting a key in order to modify it, sign or +certify it and for obtaining relevant data about a single key as a part +of other functions; when verifying a signature made by that key, for +instance. + +By default this method will select public keys, but it can select secret +keys as well. + +This first example demonstrates selecting the current key of Werner +Koch, which is due to expire at the end of 2018: + +.. code:: python + + import gpg + + fingerprint = "80615870F5BAD690333686D0F2AD85AC1E42B367" + key = gpg.Context().get_key(fingerprint) + +Whereas this example demonstrates selecting the author\'s current key +with the ``secret`` key word argument set to ``True``: + +.. code:: python + + import gpg + + fingerprint = "DB4724E6FA4286C92B4E55C4321E4E2373590E5D" + key = gpg.Context().get_key(fingerprint, secret=True) + +It is, of course, quite possible to select expired, disabled and revoked +keys with this function, but only to effectively display information +about those keys. + +It is also possible to use both unicode or string literals and byte +literals with the fingerprint when getting a key in this way. + +.. _howto-import-key: + +Importing keys +-------------- + +Importing keys is possible with the ``key_import()`` method and takes +one argument which is a bytes literal object containing either the +binary or ASCII armoured key data for one or more keys. + +The following example retrieves one or more keys from the SKS keyservers +via the web using the requests module. Since requests returns the +content as a bytes literal object, we can then use that directly to +import the resulting data into our keybox. + +.. code:: python + + import gpg + import os.path + import requests + + c = gpg.Context() + url = "https://sks-keyservers.net/pks/lookup" + pattern = input("Enter the pattern to search for key or user IDs: ") + payload = {"op": "get", "search": pattern} + + r = requests.get(url, verify=True, params=payload) + result = c.key_import(r.content) + + if result is not None and hasattr(result, "considered") is False: + print(result) + elif result is not None and hasattr(result, "considered") is True: + num_keys = len(result.imports) + new_revs = result.new_revocations + new_sigs = result.new_signatures + new_subs = result.new_sub_keys + new_uids = result.new_user_ids + new_scrt = result.secret_imported + nochange = result.unchanged + print(""" + The total number of keys considered for import was: {0} + + Number of keys revoked: {1} + Number of new signatures: {2} + Number of new subkeys: {3} + Number of new user IDs: {4} + Number of new secret keys: {5} + Number of unchanged keys: {6} + + The key IDs for all considered keys were: + """.format(num_keys, new_revs, new_sigs, new_subs, new_uids, new_scrt, + nochange)) + for i in range(num_keys): + print("{0}\n".format(result.imports[i].fpr)) + else: + pass + +NOTE: When searching for a key ID of any length or a fingerprint +(without spaces), the SKS servers require the the leading ``0x`` +indicative of hexadecimal be included. Also note that the old short key +IDs (e.g. ``0xDEADBEEF``) should no longer be used due to the relative +ease by which such key IDs can be reproduced, as demonstrated by the +Evil32 Project in 2014 (which was subsequently exploited in 2016). + +.. _import-protonmail: + +Working with ProtonMail +~~~~~~~~~~~~~~~~~~~~~~~ + +Here is a variation on the example above which checks the constrained +ProtonMail keyserver for ProtonMail public keys. + +.. code:: python + + import gpg + import requests + import sys + + print(""" + This script searches the ProtonMail key server for the specified key and + imports it. + """) + + c = gpg.Context(armor=True) + url = "https://api.protonmail.ch/pks/lookup" + ksearch = [] + + if len(sys.argv) >= 2: + keyterm = sys.argv[1] + else: + keyterm = input("Enter the key ID, UID or search string: ") + + if keyterm.count("@") == 2 and keyterm.startswith("@") is True: + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: + ksearch.append("{0}@protonmail.com".format(keyterm[1:])) + ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) + ksearch.append("{0}@pm.me".format(keyterm[1:])) + elif keyterm.count("@") == 0: + ksearch.append("{0}@protonmail.com".format(keyterm)) + ksearch.append("{0}@protonmail.ch".format(keyterm)) + ksearch.append("{0}@pm.me".format(keyterm)) + elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + elif keyterm.count("@") > 2: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + else: + ksearch.append(keyterm) + + for k in ksearch: + payload = {"op": "get", "search": k} + try: + r = requests.get(url, verify=True, params=payload) + if r.ok is True: + result = c.key_import(r.content) + elif r.ok is False: + result = r.content + except Exception as e: + result = None + + if result is not None and hasattr(result, "considered") is False: + print("{0} for {1}".format(result.decode(), k)) + elif result is not None and hasattr(result, "considered") is True: + num_keys = len(result.imports) + new_revs = result.new_revocations + new_sigs = result.new_signatures + new_subs = result.new_sub_keys + new_uids = result.new_user_ids + new_scrt = result.secret_imported + nochange = result.unchanged + print(""" + The total number of keys considered for import was: {0} + + With UIDs wholely or partially matching the following string: + + {1} + + Number of keys revoked: {2} + Number of new signatures: {3} + Number of new subkeys: {4} + Number of new user IDs: {5} + Number of new secret keys: {6} + Number of unchanged keys: {7} + + The key IDs for all considered keys were: + """.format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, + nochange)) + for i in range(num_keys): + print(result.imports[i].fpr) + print("") + elif result is None: + print(e) + +Both the above example, +`pmkey-import.py <../examples/howto/pmkey-import.py>`__, and a version +which prompts for an alternative GnuPG home directory, +`pmkey-import-alt.py <../examples/howto/pmkey-import-alt.py>`__, are +available with the other examples and are executable scripts. + +Note that while the ProtonMail servers are based on the SKS servers, +their server is related more to their API and is not feature complete by +comparison to the servers in the SKS pool. One notable difference being +that the ProtonMail server does not permit non ProtonMail users to +update their own keys, which could be a vector for attacking ProtonMail +users who may not receive a key\'s revocation if it had been +compromised. + +.. _import-hkp4py: + +Importing with HKP for Python +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Performing the same tasks with the `hkp4py +module <https://github.com/Selfnet/hkp4py>`__ (available via PyPI) is +not too much different, but does provide a number of options of benefit +to end users. Not least of which being the ability to perform some +checks on a key before importing it or not. For instance it may be the +policy of a site or project to only import keys which have not been +revoked. The hkp4py module permits such checks prior to the importing of +the keys found. + +.. code:: python + + import gpg + import hkp4py + import sys + + c = gpg.Context() + server = hkp4py.KeyServer("hkps://hkps.pool.sks-keyservers.net") + results = [] + + if len(sys.argv) > 2: + pattern = " ".join(sys.argv[1:]) + elif len(sys.argv) == 2: + pattern = sys.argv[1] + else: + pattern = input("Enter the pattern to search for keys or user IDs: ") + + try: + keys = server.search(pattern) + print("Found {0} key(s).".format(len(keys))) + except Exception as e: + keys = [] + for logrus in pattern.split(): + if logrus.startswith("0x") is True: + key = server.search(logrus) + else: + key = server.search("0x{0}".format(logrus)) + keys.append(key[0]) + print("Found {0} key(s).".format(len(keys))) + + for key in keys: + import_result = c.key_import(key.key_blob) + results.append(import_result) + + for result in results: + if result is not None and hasattr(result, "considered") is False: + print(result) + elif result is not None and hasattr(result, "considered") is True: + num_keys = len(result.imports) + new_revs = result.new_revocations + new_sigs = result.new_signatures + new_subs = result.new_sub_keys + new_uids = result.new_user_ids + new_scrt = result.secret_imported + nochange = result.unchanged + print(""" + The total number of keys considered for import was: {0} + + Number of keys revoked: {1} + Number of new signatures: {2} + Number of new subkeys: {3} + Number of new user IDs: {4} + Number of new secret keys: {5} + Number of unchanged keys: {6} + + The key IDs for all considered keys were: + """.format(num_keys, new_revs, new_sigs, new_subs, new_uids, new_scrt, + nochange)) + for i in range(num_keys): + print(result.imports[i].fpr) + print("") + else: + pass + +Since the hkp4py module handles multiple keys just as effectively as one +(``keys`` is a list of responses per matching key), the example above is +able to do a little bit more with the returned data before anything is +actually imported. + +.. _import-protonmail-hkp4py: + +Importing from ProtonMail with HKP for Python +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Though this can provide certain benefits even when working with +ProtonMail, the scope is somewhat constrained there due to the +limitations of the ProtonMail keyserver. + +For instance, searching the SKS keyserver pool for the term \"gnupg\" +produces hundreds of results from any time the word appears in any part +of a user ID. Performing the same search on the ProtonMail keyserver +returns zero results, even though there are at least two test accounts +which include it as part of the username. + +The cause of this discrepancy is the deliberate configuration of that +server by ProtonMail to require an exact match of the full email address +of the ProtonMail user whose key is being requested. Presumably this is +intended to reduce breaches of privacy of their users as an email +address must already be known before a key for that address can be +obtained. + +#. Import from ProtonMail via HKP for Python Example no. 1 + + The following script is avalable with the rest of the examples under + the somewhat less than original name, ``pmkey-import-hkp.py``. + + .. code:: python + + import gpg + import hkp4py + import os.path + import sys + + print(""" + This script searches the ProtonMail key server for the specified key and + imports it. + + Usage: pmkey-import-hkp.py [search strings] + """) + + c = gpg.Context(armor=True) + server = hkp4py.KeyServer("hkps://api.protonmail.ch") + keyterms = [] + ksearch = [] + allkeys = [] + results = [] + paradox = [] + homeless = None + + if len(sys.argv) > 2: + keyterms = sys.argv[1:] + elif len(sys.argv) == 2: + keyterm = sys.argv[1] + keyterms.append(keyterm) + else: + key_term = input("Enter the key ID, UID or search string: ") + keyterms = key_term.split() + + for keyterm in keyterms: + if keyterm.count("@") == 2 and keyterm.startswith("@") is True: + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: + ksearch.append("{0}@protonmail.com".format(keyterm[1:])) + ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) + ksearch.append("{0}@pm.me".format(keyterm[1:])) + elif keyterm.count("@") == 0: + ksearch.append("{0}@protonmail.com".format(keyterm)) + ksearch.append("{0}@protonmail.ch".format(keyterm)) + ksearch.append("{0}@pm.me".format(keyterm)) + elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + elif keyterm.count("@") > 2: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + else: + ksearch.append(keyterm) + + for k in ksearch: + print("Checking for key for: {0}".format(k)) + try: + keys = server.search(k) + if isinstance(keys, list) is True: + for key in keys: + allkeys.append(key) + try: + import_result = c.key_import(key.key_blob) + except Exception as e: + import_result = c.key_import(key.key) + else: + paradox.append(keys) + import_result = None + except Exception as e: + import_result = None + results.append(import_result) + + for result in results: + if result is not None and hasattr(result, "considered") is False: + print("{0} for {1}".format(result.decode(), k)) + elif result is not None and hasattr(result, "considered") is True: + num_keys = len(result.imports) + new_revs = result.new_revocations + new_sigs = result.new_signatures + new_subs = result.new_sub_keys + new_uids = result.new_user_ids + new_scrt = result.secret_imported + nochange = result.unchanged + print(""" + The total number of keys considered for import was: {0} + + With UIDs wholely or partially matching the following string: + + {1} + + Number of keys revoked: {2} + Number of new signatures: {3} + Number of new subkeys: {4} + Number of new user IDs: {5} + Number of new secret keys: {6} + Number of unchanged keys: {7} + + The key IDs for all considered keys were: + """.format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, + nochange)) + for i in range(num_keys): + print(result.imports[i].fpr) + print("") + elif result is None: + pass + +#. Import from ProtonMail via HKP for Python Example no. 2 + + Like its counterpart above, this script can also be found with the + rest of the examples, by the name pmkey-import-hkp-alt.py. + + With this script a modicum of effort has been made to treat anything + passed as a ``homedir`` which either does not exist or which is not a + directory, as also being a pssible user ID to check for. It\'s not + guaranteed to pick up on all such cases, but it should cover most of + them. + + .. code:: python + + import gpg + import hkp4py + import os.path + import sys + + print(""" + This script searches the ProtonMail key server for the specified key and + imports it. Optionally enables specifying a different GnuPG home directory. + + Usage: pmkey-import-hkp.py [homedir] [search string] + or: pmkey-import-hkp.py [search string] + """) + + c = gpg.Context(armor=True) + server = hkp4py.KeyServer("hkps://api.protonmail.ch") + keyterms = [] + ksearch = [] + allkeys = [] + results = [] + paradox = [] + homeless = None + + if len(sys.argv) > 3: + homedir = sys.argv[1] + keyterms = sys.argv[2:] + elif len(sys.argv) == 3: + homedir = sys.argv[1] + keyterm = sys.argv[2] + keyterms.append(keyterm) + elif len(sys.argv) == 2: + homedir = "" + keyterm = sys.argv[1] + keyterms.append(keyterm) + else: + keyterm = input("Enter the key ID, UID or search string: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + keyterms.append(keyterm) + + if len(homedir) == 0: + homedir = None + homeless = False + + if homedir is not None: + if homedir.startswith("~"): + if os.path.exists(os.path.expanduser(homedir)) is True: + if os.path.isdir(os.path.expanduser(homedir)) is True: + c.home_dir = os.path.realpath(os.path.expanduser(homedir)) + else: + homeless = True + else: + homeless = True + elif os.path.exists(os.path.realpath(homedir)) is True: + if os.path.isdir(os.path.realpath(homedir)) is True: + c.home_dir = os.path.realpath(homedir) + else: + homeless = True + else: + homeless = True + + # First check to see if the homedir really is a homedir and if not, treat it as + # a search string. + if homeless is True: + keyterms.append(homedir) + c.home_dir = None + else: + pass + + for keyterm in keyterms: + if keyterm.count("@") == 2 and keyterm.startswith("@") is True: + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + ksearch.append(keyterm[1:]) + elif keyterm.count("@") == 1 and keyterm.startswith("@") is True: + ksearch.append("{0}@protonmail.com".format(keyterm[1:])) + ksearch.append("{0}@protonmail.ch".format(keyterm[1:])) + ksearch.append("{0}@pm.me".format(keyterm[1:])) + elif keyterm.count("@") == 0: + ksearch.append("{0}@protonmail.com".format(keyterm)) + ksearch.append("{0}@protonmail.ch".format(keyterm)) + ksearch.append("{0}@pm.me".format(keyterm)) + elif keyterm.count("@") == 2 and keyterm.startswith("@") is False: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + elif keyterm.count("@") > 2: + uidlist = keyterm.split("@") + for uid in uidlist: + ksearch.append("{0}@protonmail.com".format(uid)) + ksearch.append("{0}@protonmail.ch".format(uid)) + ksearch.append("{0}@pm.me".format(uid)) + else: + ksearch.append(keyterm) + + for k in ksearch: + print("Checking for key for: {0}".format(k)) + try: + keys = server.search(k) + if isinstance(keys, list) is True: + for key in keys: + allkeys.append(key) + try: + import_result = c.key_import(key.key_blob) + except Exception as e: + import_result = c.key_import(key.key) + else: + paradox.append(keys) + import_result = None + except Exception as e: + import_result = None + results.append(import_result) + + for result in results: + if result is not None and hasattr(result, "considered") is False: + print("{0} for {1}".format(result.decode(), k)) + elif result is not None and hasattr(result, "considered") is True: + num_keys = len(result.imports) + new_revs = result.new_revocations + new_sigs = result.new_signatures + new_subs = result.new_sub_keys + new_uids = result.new_user_ids + new_scrt = result.secret_imported + nochange = result.unchanged + print(""" + The total number of keys considered for import was: {0} + + With UIDs wholely or partially matching the following string: + + {1} + + Number of keys revoked: {2} + Number of new signatures: {3} + Number of new subkeys: {4} + Number of new user IDs: {5} + Number of new secret keys: {6} + Number of unchanged keys: {7} + + The key IDs for all considered keys were: + """.format(num_keys, k, new_revs, new_sigs, new_subs, new_uids, new_scrt, + nochange)) + for i in range(num_keys): + print(result.imports[i].fpr) + print("") + elif result is None: + pass + +.. _howto-export-key: + +Exporting keys +-------------- + +Exporting keys remains a reasonably simple task, but has been separated +into three different functions for the OpenPGP cryptographic engine. Two +of those functions are for exporting public keys and the third is for +exporting secret keys. + +.. _howto-export-public-key: + +Exporting public keys +~~~~~~~~~~~~~~~~~~~~~ + +There are two methods of exporting public keys, both of which are very +similar to the other. The default method, ``key_export()``, will export +a public key or keys matching a specified pattern as normal. The +alternative, the ``key_export_minimal()`` method, will do the same thing +except producing a minimised output with extra signatures and third +party signatures or certifications removed. + +.. code:: python + + import gpg + import os.path + import sys + + print(""" + This script exports one or more public keys. + """) + + c = gpg.Context(armor=True) + + if len(sys.argv) >= 4: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = sys.argv[3] + elif len(sys.argv) == 3: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = input("Enter the GPG configuration directory path (optional): ") + elif len(sys.argv) == 2: + keyfile = sys.argv[1] + logrus = input("Enter the UID matching the key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + else: + keyfile = input("Enter the path and filename to save the secret key to: ") + logrus = input("Enter the UID matching the key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + + if homedir.startswith("~"): + if os.path.exists(os.path.expanduser(homedir)) is True: + c.home_dir = os.path.expanduser(homedir) + else: + pass + elif os.path.exists(homedir) is True: + c.home_dir = homedir + else: + pass + + try: + result = c.key_export(pattern=logrus) + except: + result = c.key_export(pattern=None) + + if result is not None: + with open(keyfile, "wb") as f: + f.write(result) + else: + pass + +It should be noted that the result will only return ``None`` when a +search pattern has been entered, but has not matched any keys. When the +search pattern itself is set to ``None`` this triggers the exporting of +the entire public keybox. + +.. code:: python + + import gpg + import os.path + import sys + + print(""" + This script exports one or more public keys in minimised form. + """) + + c = gpg.Context(armor=True) + + if len(sys.argv) >= 4: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = sys.argv[3] + elif len(sys.argv) == 3: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = input("Enter the GPG configuration directory path (optional): ") + elif len(sys.argv) == 2: + keyfile = sys.argv[1] + logrus = input("Enter the UID matching the key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + else: + keyfile = input("Enter the path and filename to save the secret key to: ") + logrus = input("Enter the UID matching the key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + + if homedir.startswith("~"): + if os.path.exists(os.path.expanduser(homedir)) is True: + c.home_dir = os.path.expanduser(homedir) + else: + pass + elif os.path.exists(homedir) is True: + c.home_dir = homedir + else: + pass + + try: + result = c.key_export_minimal(pattern=logrus) + except: + result = c.key_export_minimal(pattern=None) + + if result is not None: + with open(keyfile, "wb") as f: + f.write(result) + else: + pass + +.. _howto-export-secret-key: + +Exporting secret keys +~~~~~~~~~~~~~~~~~~~~~ + +Exporting secret keys is, functionally, very similar to exporting public +keys; save for the invocation of ``pinentry`` via ``gpg-agent`` in order +to securely enter the key\'s passphrase and authorise the export. + +The following example exports the secret key to a file which is then set +with the same permissions as the output files created by the command +line secret key export options. + +.. code:: python + + import gpg + import os + import os.path + import sys + + print(""" + This script exports one or more secret keys. + + The gpg-agent and pinentry are invoked to authorise the export. + """) + + c = gpg.Context(armor=True) + + if len(sys.argv) >= 4: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = sys.argv[3] + elif len(sys.argv) == 3: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = input("Enter the GPG configuration directory path (optional): ") + elif len(sys.argv) == 2: + keyfile = sys.argv[1] + logrus = input("Enter the UID matching the secret key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + else: + keyfile = input("Enter the path and filename to save the secret key to: ") + logrus = input("Enter the UID matching the secret key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + + if len(homedir) == 0: + homedir = None + elif homedir.startswith("~"): + userdir = os.path.expanduser(homedir) + if os.path.exists(userdir) is True: + homedir = os.path.realpath(userdir) + else: + homedir = None + else: + homedir = os.path.realpath(homedir) + + if os.path.exists(homedir) is False: + homedir = None + else: + if os.path.isdir(homedir) is False: + homedir = None + else: + pass + + if homedir is not None: + c.home_dir = homedir + else: + pass + + try: + result = c.key_export_secret(pattern=logrus) + except: + result = c.key_export_secret(pattern=None) + + if result is not None: + with open(keyfile, "wb") as f: + f.write(result) + os.chmod(keyfile, 0o600) + else: + pass + +Alternatively the approach of the following script can be used. This +longer example saves the exported secret key(s) in files in the GnuPG +home directory, in addition to setting the file permissions as only +readable and writable by the user. It also exports the secret key(s) +twice in order to output both GPG binary (``.gpg``) and ASCII armoured +(``.asc``) files. + +.. code:: python + + import gpg + import os + import os.path + import subprocess + import sys + + print(""" + This script exports one or more secret keys as both ASCII armored and binary + file formats, saved in files within the user's GPG home directory. + + The gpg-agent and pinentry are invoked to authorise the export. + """) + + if sys.platform == "win32": + gpgconfcmd = "gpgconf.exe --list-dirs homedir" + else: + gpgconfcmd = "gpgconf --list-dirs homedir" + + a = gpg.Context(armor=True) + b = gpg.Context() + c = gpg.Context() + + if len(sys.argv) >= 4: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = sys.argv[3] + elif len(sys.argv) == 3: + keyfile = sys.argv[1] + logrus = sys.argv[2] + homedir = input("Enter the GPG configuration directory path (optional): ") + elif len(sys.argv) == 2: + keyfile = sys.argv[1] + logrus = input("Enter the UID matching the secret key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + else: + keyfile = input("Enter the filename to save the secret key to: ") + logrus = input("Enter the UID matching the secret key(s) to export: ") + homedir = input("Enter the GPG configuration directory path (optional): ") + + if len(homedir) == 0: + homedir = None + elif homedir.startswith("~"): + userdir = os.path.expanduser(homedir) + if os.path.exists(userdir) is True: + homedir = os.path.realpath(userdir) + else: + homedir = None + else: + homedir = os.path.realpath(homedir) + + if os.path.exists(homedir) is False: + homedir = None + else: + if os.path.isdir(homedir) is False: + homedir = None + else: + pass + + if homedir is not None: + c.home_dir = homedir + else: + pass + + if c.home_dir is not None: + if c.home_dir.endswith("/"): + gpgfile = "{0}{1}.gpg".format(c.home_dir, keyfile) + ascfile = "{0}{1}.asc".format(c.home_dir, keyfile) + else: + gpgfile = "{0}/{1}.gpg".format(c.home_dir, keyfile) + ascfile = "{0}/{1}.asc".format(c.home_dir, keyfile) + else: + if os.path.exists(os.environ["GNUPGHOME"]) is True: + hd = os.environ["GNUPGHOME"] + else: + try: + hd = subprocess.getoutput(gpgconfcmd) + except: + process = subprocess.Popen(gpgconfcmd.split(), + stdout=subprocess.PIPE) + procom = process.communicate() + if sys.version_info[0] == 2: + hd = procom[0].strip() + else: + hd = procom[0].decode().strip() + gpgfile = "{0}/{1}.gpg".format(hd, keyfile) + ascfile = "{0}/{1}.asc".format(hd, keyfile) + + try: + a_result = a.key_export_secret(pattern=logrus) + b_result = b.key_export_secret(pattern=logrus) + except: + a_result = a.key_export_secret(pattern=None) + b_result = b.key_export_secret(pattern=None) + + if a_result is not None: + with open(ascfile, "wb") as f: + f.write(a_result) + os.chmod(ascfile, 0o600) + else: + pass + + if b_result is not None: + with open(gpgfile, "wb") as f: + f.write(b_result) + os.chmod(gpgfile, 0o600) + else: + pass + +.. _howto-send-public-key: + +Sending public keys to the SKS Keyservers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As with the previous section on importing keys, the ``hkp4py`` module +adds another option with exporting keys in order to send them to the +public keyservers. + +The following example demonstrates how this may be done. + +.. code:: python + + import gpg + import hkp4py + import os.path + import sys + + print(""" + This script sends one or more public keys to the SKS keyservers and is + essentially a slight variation on the export-key.py script. + """) + + c = gpg.Context(armor=True) + server = hkp4py.KeyServer("hkps://hkps.pool.sks-keyservers.net") + + if len(sys.argv) > 2: + logrus = " ".join(sys.argv[1:]) + elif len(sys.argv) == 2: + logrus = sys.argv[1] + else: + logrus = input("Enter the UID matching the key(s) to send: ") + + if len(logrus) > 0: + try: + export_result = c.key_export(pattern=logrus) + except Exception as e: + print(e) + export_result = None + else: + export_result = c.key_export(pattern=None) + + if export_result is not None: + try: + try: + send_result = server.add(export_result) + except: + send_result = server.add(export_result.decode()) + if send_result is not None: + print(send_result) + else: + pass + except Exception as e: + print(e) + else: + pass + +An expanded version of this script with additional functions for +specifying an alternative homedir location is in the examples directory +as ``send-key-to-keyserver.py``. + +The ``hkp4py`` module appears to handle both string and byte literal +text data equally well, but the GPGME bindings deal primarily with byte +literal data only and so this script sends in that format first, then +tries the string literal form. + +.. _howto-the-basics: + +Basic Functions +=============== + +The most frequently called features of any cryptographic library will be +the most fundamental tasks for encryption software. In this section we +will look at how to programmatically encrypt data, decrypt it, sign it +and verify signatures. + +.. _howto-basic-encryption: + +Encryption +---------- + +Encrypting is very straight forward. In the first example below the +message, ``text``, is encrypted to a single recipient\'s key. In the +second example the message will be encrypted to multiple recipients. + +.. _howto-basic-encryption-single: + +Encrypting to one key +~~~~~~~~~~~~~~~~~~~~~ + +Once the the Context is set the main issues with encrypting data is +essentially reduced to key selection and the keyword arguments specified +in the ``gpg.Context().encrypt()`` method. + +Those keyword arguments are: ``recipients``, a list of keys encrypted to +(covered in greater detail in the following section); ``sign``, whether +or not to sign the plaintext data, see subsequent sections on signing +and verifying signatures below (defaults to ``True``); ``sink``, to +write results or partial results to a secure sink instead of returning +it (defaults to ``None``); ``passphrase``, only used when utilising +symmetric encryption (defaults to ``None``); ``always_trust``, used to +override the trust model settings for recipient keys (defaults to +``False``); ``add_encrypt_to``, utilises any preconfigured +``encrypt-to`` or ``default-key`` settings in the user\'s ``gpg.conf`` +file (defaults to ``False``); ``prepare``, prepare for encryption +(defaults to ``False``); ``expect_sign``, prepare for signing (defaults +to ``False``); ``compress``, compresses the plaintext prior to +encryption (defaults to ``True``). + +.. code:: python + + import gpg + + a_key = "0x12345678DEADBEEF" + text = b"""Some text to test with. + + Since the text in this case must be bytes, it is most likely that + the input form will be a separate file which is opened with "rb" + as this is the simplest method of obtaining the correct data format. + """ + + c = gpg.Context(armor=True) + rkey = list(c.keylist(pattern=a_key, secret=False)) + ciphertext, result, sign_result = c.encrypt(text, recipients=rkey, sign=False) + + with open("secret_plans.txt.asc", "wb") as afile: + afile.write(ciphertext) + +Though this is even more likely to be used like this; with the plaintext +input read from a file, the recipient keys used for encryption +regardless of key trust status and the encrypted output also encrypted +to any preconfigured keys set in the ``gpg.conf`` file: + +.. code:: python + + import gpg + + a_key = "0x12345678DEADBEEF" + + with open("secret_plans.txt", "rb") as afile: + text = afile.read() + + c = gpg.Context(armor=True) + rkey = list(c.keylist(pattern=a_key, secret=False)) + ciphertext, result, sign_result = c.encrypt(text, recipients=rkey, sign=True, + always_trust=True, + add_encrypt_to=True) + + with open("secret_plans.txt.asc", "wb") as afile: + afile.write(ciphertext) + +If the ``recipients`` paramater is empty then the plaintext is encrypted +symmetrically. If no ``passphrase`` is supplied as a parameter or via a +callback registered with the ``Context()`` then an out-of-band prompt +for the passphrase via pinentry will be invoked. + +.. _howto-basic-encryption-multiple: + +Encrypting to multiple keys +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Encrypting to multiple keys essentially just expands upon the key +selection process and the recipients from the previous examples. + +The following example encrypts a message (``text``) to everyone with an +email address on the ``gnupg.org`` domain, [5]_ but does *not* encrypt +to a default key or other key which is configured to normally encrypt +to. + +.. code:: python + + import gpg + + text = b"""Oh look, another test message. + + The same rules apply as with the previous example and more likely + than not, the message will actually be drawn from reading the + contents of a file or, maybe, from entering data at an input() + prompt. + + Since the text in this case must be bytes, it is most likely that + the input form will be a separate file which is opened with "rb" + as this is the simplest method of obtaining the correct data + format. + """ + + c = gpg.Context(armor=True) + rpattern = list(c.keylist(pattern="@gnupg.org", secret=False)) + logrus = [] + + for i in range(len(rpattern)): + if rpattern[i].can_encrypt == 1: + logrus.append(rpattern[i]) + + ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, + sign=False, always_trust=True) + + with open("secret_plans.txt.asc", "wb") as afile: + afile.write(ciphertext) + +All it would take to change the above example to sign the message and +also encrypt the message to any configured default keys would be to +change the ``c.encrypt`` line to this: + +.. code:: python + + ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, + always_trust=True, + add_encrypt_to=True) + +The only keyword arguments requiring modification are those for which +the default values are changing. The default value of ``sign`` is +``True``, the default of ``always_trust`` is ``False``, the default of +``add_encrypt_to`` is ``False``. + +If ``always_trust`` is not set to ``True`` and any of the recipient keys +are not trusted (e.g. not signed or locally signed) then the encryption +will raise an error. It is possible to mitigate this somewhat with +something more like this: + +.. code:: python + + import gpg + + with open("secret_plans.txt.asc", "rb") as afile: + text = afile.read() + + c = gpg.Context(armor=True) + rpattern = list(c.keylist(pattern="@gnupg.org", secret=False)) + logrus = [] + + for i in range(len(rpattern)): + if rpattern[i].can_encrypt == 1: + logrus.append(rpattern[i]) + + try: + ciphertext, result, sign_result = c.encrypt(text, recipients=logrus, + add_encrypt_to=True) + except gpg.errors.InvalidRecipients as e: + for i in range(len(e.recipients)): + for n in range(len(logrus)): + if logrus[n].fpr == e.recipients[i].fpr: + logrus.remove(logrus[n]) + else: + pass + try: + ciphertext, result, sign_result = c.encrypt(text, + recipients=logrus, + add_encrypt_to=True) + with open("secret_plans.txt.asc", "wb") as afile: + afile.write(ciphertext) + except: + pass + +This will attempt to encrypt to all the keys searched for, then remove +invalid recipients if it fails and try again. + +.. _howto-basic-decryption: + +Decryption +---------- + +Decrypting something encrypted to a key in one\'s secret keyring is +fairly straight forward. + +In this example code, however, preconfiguring either ``gpg.Context()`` +or ``gpg.core.Context()`` as ``c`` is unnecessary because there is no +need to modify the Context prior to conducting the decryption and since +the Context is only used once, setting it to ``c`` simply adds lines for +no gain. + +.. code:: python + + import gpg + + ciphertext = input("Enter path and filename of encrypted file: ") + newfile = input("Enter path and filename of file to save decrypted data to: ") + + with open(ciphertext, "rb") as cfile: + try: + plaintext, result, verify_result = gpg.Context().decrypt(cfile) + except gpg.errors.GPGMEError as e: + plaintext = None + print(e) + + if plaintext is not None: + with open(newfile, "wb") as nfile: + nfile.write(plaintext) + else: + pass + +The data available in ``plaintext`` in this example is the decrypted +content as a byte object, the recipient key IDs and algorithms in +``result`` and the results of verifying any signatures of the data in +``verify_result``. + +.. _howto-basic-signing: + +Signing text and files +---------------------- + +The following sections demonstrate how to specify keys to sign with. + +.. _howto-basic-signing-signers: + +Signing key selection +~~~~~~~~~~~~~~~~~~~~~ + +By default GPGME and the Python bindings will use the default key +configured for the user invoking the GPGME API. If there is no default +key specified and there is more than one secret key available it may be +necessary to specify the key or keys with which to sign messages and +files. + +.. code:: python + + import gpg + + logrus = input("Enter the email address or string to match signing keys to: ") + hancock = gpg.Context().keylist(pattern=logrus, secret=True) + sig_src = list(hancock) + +The signing examples in the following sections include the explicitly +designated ``signers`` parameter in two of the five examples; once where +the resulting signature would be ASCII armoured and once where it would +not be armoured. + +While it would be possible to enter a key ID or fingerprint here to +match a specific key, it is not possible to enter two fingerprints and +match two keys since the patten expects a string, bytes or None and not +a list. A string with two fingerprints won\'t match any single key. + +.. _howto-basic-signing-normal: + +Normal or default signing messages or files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The normal or default signing process is essentially the same as is most +often invoked when also encrypting a message or file. So when the +encryption component is not utilised, the result is to produce an +encoded and signed output which may or may not be ASCII armoured and +which may or may not also be compressed. + +By default compression will be used unless GnuPG detects that the +plaintext is already compressed. ASCII armouring will be determined +according to the value of ``gpg.Context().armor``. + +The compression algorithm is selected in much the same way as the +symmetric encryption algorithm or the hash digest algorithm is when +multiple keys are involved; from the preferences saved into the key +itself or by comparison with the preferences with all other keys +involved. + +.. code:: python + + import gpg + + text0 = """Declaration of ... something. + + """ + text = text0.encode() + + c = gpg.Context(armor=True, signers=sig_src) + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.NORMAL) + + with open("/path/to/statement.txt.asc", "w") as afile: + afile.write(signed_data.decode()) + +Though everything in this example is accurate, it is more likely that +reading the input data from another file and writing the result to a new +file will be performed more like the way it is done in the next example. +Even if the output format is ASCII armoured. + +.. code:: python + + import gpg + + with open("/path/to/statement.txt", "rb") as tfile: + text = tfile.read() + + c = gpg.Context() + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.NORMAL) + + with open("/path/to/statement.txt.sig", "wb") as afile: + afile.write(signed_data) + +.. _howto-basic-signing-detached: + +Detached signing messages and files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Detached signatures will often be needed in programmatic uses of GPGME, +either for signing files (e.g. tarballs of code releases) or as a +component of message signing (e.g. PGP/MIME encoded email). + +.. code:: python + + import gpg + + text0 = """Declaration of ... something. + + """ + text = text0.encode() + + c = gpg.Context(armor=True) + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.DETACH) + + with open("/path/to/statement.txt.asc", "w") as afile: + afile.write(signed_data.decode()) + +As with normal signatures, detached signatures are best handled as byte +literals, even when the output is ASCII armoured. + +.. code:: python + + import gpg + + with open("/path/to/statement.txt", "rb") as tfile: + text = tfile.read() + + c = gpg.Context(signers=sig_src) + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.DETACH) + + with open("/path/to/statement.txt.sig", "wb") as afile: + afile.write(signed_data) + +.. _howto-basic-signing-clear: + +Clearsigning messages or text +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Though PGP/in-line messages are no longer encouraged in favour of +PGP/MIME, there is still sometimes value in utilising in-line +signatures. This is where clear-signed messages or text is of value. + +.. code:: python + + import gpg + + text0 = """Declaration of ... something. + + """ + text = text0.encode() + + c = gpg.Context() + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.CLEAR) + + with open("/path/to/statement.txt.asc", "w") as afile: + afile.write(signed_data.decode()) + +In spite of the appearance of a clear-signed message, the data handled +by GPGME in signing it must still be byte literals. + +.. code:: python + + import gpg + + with open("/path/to/statement.txt", "rb") as tfile: + text = tfile.read() + + c = gpg.Context() + signed_data, result = c.sign(text, mode=gpg.constants.sig.mode.CLEAR) + + with open("/path/to/statement.txt.asc", "wb") as afile: + afile.write(signed_data) + +.. _howto-basic-verification: + +Signature verification +---------------------- + +Essentially there are two principal methods of verification of a +signature. The first of these is for use with the normal or default +signing method and for clear-signed messages. The second is for use with +files and data with detached signatures. + +The following example is intended for use with the default signing +method where the file was not ASCII armoured: + +.. code:: python + + import gpg + import time + + filename = "statement.txt" + gpg_file = "statement.txt.gpg" + + c = gpg.Context() + + try: + data, result = c.verify(open(gpg_file)) + verified = True + except gpg.errors.BadSignatures as e: + verified = False + print(e) + + if verified is True: + for i in range(len(result.signatures)): + sign = result.signatures[i] + print("""Good signature from: + {0} + with key {1} + made at {2} + """.format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, + time.ctime(sign.timestamp))) + else: + pass + +Whereas this next example, which is almost identical would work with +normal ASCII armoured files and with clear-signed files: + +.. code:: python + + import gpg + import time + + filename = "statement.txt" + asc_file = "statement.txt.asc" + + c = gpg.Context() + + try: + data, result = c.verify(open(asc_file)) + verified = True + except gpg.errors.BadSignatures as e: + verified = False + print(e) + + if verified is True: + for i in range(len(result.signatures)): + sign = result.signatures[i] + print("""Good signature from: + {0} + with key {1} + made at {2} + """.format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, + time.ctime(sign.timestamp))) + else: + pass + +In both of the previous examples it is also possible to compare the +original data that was signed against the signed data in ``data`` to see +if it matches with something like this: + +.. code:: python + + with open(filename, "rb") as afile: + text = afile.read() + + if text == data: + print("Good signature.") + else: + pass + +The following two examples, however, deal with detached signatures. With +his method of verification the data that was signed does not get +returned since it is already being explicitly referenced in the first +argument of ``c.verify``. So ``data`` is ``None`` and only the +information in ``result`` is available. + +.. code:: python + + import gpg + import time + + filename = "statement.txt" + sig_file = "statement.txt.sig" + + c = gpg.Context() + + try: + data, result = c.verify(open(filename), open(sig_file)) + verified = True + except gpg.errors.BadSignatures as e: + verified = False + print(e) + + if verified is True: + for i in range(len(result.signatures)): + sign = result.signatures[i] + print("""Good signature from: + {0} + with key {1} + made at {2} + """.format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, + time.ctime(sign.timestamp))) + else: + pass + +.. code:: python + + import gpg + import time + + filename = "statement.txt" + asc_file = "statement.txt.asc" + + c = gpg.Context() + + try: + data, result = c.verify(open(filename), open(asc_file)) + verified = True + except gpg.errors.BadSignatures as e: + verified = False + print(e) + + if verified is True: + for i in range(len(result.signatures)): + sign = result.signatures[i] + print("""Good signature from: + {0} + with key {1} + made at {2} + """.format(c.get_key(sign.fpr).uids[0].uid, sign.fpr, + time.ctime(sign.timestamp))) + else: + pass + +.. _key-generation: + +Creating keys and subkeys +========================= + +The one thing, aside from GnuPG itself, that GPGME depends on, of +course, is the keys themselves. So it is necessary to be able to +generate them and modify them by adding subkeys, revoking or disabling +them, sometimes deleting them and doing the same for user IDs. + +In the following examples a key will be created for the world\'s +greatest secret agent, Danger Mouse. Since Danger Mouse is a secret +agent he needs to be able to protect information to ``SECRET`` level +clearance, so his keys will be 3072-bit keys. + +The pre-configured ``gpg.conf`` file which sets cipher, digest and other +preferences contains the following configuration parameters: + +.. code:: conf + + expert + allow-freeform-uid + allow-secret-key-import + trust-model tofu+pgp + tofu-default-policy unknown + enable-large-rsa + enable-dsa2 + cert-digest-algo SHA512 + default-preference-list TWOFISH CAMELLIA256 AES256 CAMELLIA192 AES192 CAMELLIA128 AES BLOWFISH IDEA CAST5 3DES SHA512 SHA384 SHA256 SHA224 RIPEMD160 SHA1 ZLIB BZIP2 ZIP Uncompressed + personal-cipher-preferences TWOFISH CAMELLIA256 AES256 CAMELLIA192 AES192 CAMELLIA128 AES BLOWFISH IDEA CAST5 3DES + personal-digest-preferences SHA512 SHA384 SHA256 SHA224 RIPEMD160 SHA1 + personal-compress-preferences ZLIB BZIP2 ZIP Uncompressed + +.. _keygen-primary: + +Primary key +----------- + +Generating a primary key uses the ``create_key`` method in a Context. It +contains multiple arguments and keyword arguments, including: +``userid``, ``algorithm``, ``expires_in``, ``expires``, ``sign``, +``encrypt``, ``certify``, ``authenticate``, ``passphrase`` and +``force``. The defaults for all of those except ``userid``, +``algorithm``, ``expires_in``, ``expires`` and ``passphrase`` is +``False``. The defaults for ``algorithm`` and ``passphrase`` is +``None``. The default for ``expires_in`` is ``0``. The default for +``expires`` is ``True``. There is no default for ``userid``. + +If ``passphrase`` is left as ``None`` then the key will not be generated +with a passphrase, if ``passphrase`` is set to a string then that will +be the passphrase and if ``passphrase`` is set to ``True`` then +gpg-agent will launch pinentry to prompt for a passphrase. For the sake +of convenience, these examples will keep ``passphrase`` set to ``None``. + +.. code:: python + + import gpg + + c = gpg.Context() + + c.home_dir = "~/.gnupg-dm" + userid = "Danger Mouse <[email protected]>" + + dmkey = c.create_key(userid, algorithm="rsa3072", expires_in=31536000, + sign=True, certify=True) + +One thing to note here is the use of setting the ``c.home_dir`` +parameter. This enables generating the key or keys in a different +location. In this case to keep the new key data created for this example +in a separate location rather than adding it to existing and active key +store data. As with the default directory, ``~/.gnupg``, any temporary +or separate directory needs the permissions set to only permit access by +the directory owner. On posix systems this means setting the directory +permissions to 700. + +The ``temp-homedir-config.py`` script in the HOWTO examples directory +will create an alternative homedir with these configuration options +already set and the correct directory and file permissions. + +The successful generation of the key can be confirmed via the returned +``GenkeyResult`` object, which includes the following data: + +.. code:: python + + print(""" + Fingerprint: {0} + Primary Key: {1} + Public Key: {2} + Secret Key: {3} + Sub Key: {4} + User IDs: {5} + """.format(dmkey.fpr, dmkey.primary, dmkey.pubkey, dmkey.seckey, dmkey.sub, + dmkey.uid)) + +Alternatively the information can be confirmed using the command line +program: + +.. code:: shell + + bash-4.4$ gpg --homedir ~/.gnupg-dm -K + ~/.gnupg-dm/pubring.kbx + ---------------------- + sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] + 177B7C25DB99745EE2EE13ED026D2F19E99E63AA + uid [ultimate] Danger Mouse <[email protected]> + + bash-4.4$ + +As with generating keys manually, to preconfigure expanded preferences +for the cipher, digest and compression algorithms, the ``gpg.conf`` file +must contain those details in the home directory in which the new key is +being generated. I used a cut down version of my own ``gpg.conf`` file +in order to be able to generate this: + +.. code:: shell + + bash-4.4$ gpg --homedir ~/.gnupg-dm --edit-key 177B7C25DB99745EE2EE13ED026D2F19E99E63AA showpref quit + Secret key is available. + + sec rsa3072/026D2F19E99E63AA + created: 2018-03-15 expires: 2019-03-15 usage: SC + trust: ultimate validity: ultimate + [ultimate] (1). Danger Mouse <[email protected]> + + [ultimate] (1). Danger Mouse <[email protected]> + Cipher: TWOFISH, CAMELLIA256, AES256, CAMELLIA192, AES192, CAMELLIA128, AES, BLOWFISH, IDEA, CAST5, 3DES + Digest: SHA512, SHA384, SHA256, SHA224, RIPEMD160, SHA1 + Compression: ZLIB, BZIP2, ZIP, Uncompressed + Features: MDC, Keyserver no-modify + + bash-4.4$ + +.. _keygen-subkeys: + +Subkeys +------- + +Adding subkeys to a primary key is fairly similar to creating the +primary key with the ``create_subkey`` method. Most of the arguments are +the same, but not quite all. Instead of the ``userid`` argument there is +now a ``key`` argument for selecting which primary key to add the subkey +to. + +In the following example an encryption subkey will be added to the +primary key. Since Danger Mouse is a security conscious secret agent, +this subkey will only be valid for about six months, half the length of +the primary key. + +.. code:: python + + import gpg + + c = gpg.Context() + c.home_dir = "~/.gnupg-dm" + + key = c.get_key(dmkey.fpr, secret=True) + dmsub = c.create_subkey(key, algorithm="rsa3072", expires_in=15768000, + encrypt=True) + +As with the primary key, the results here can be checked with: + +.. code:: python + + print(""" + Fingerprint: {0} + Primary Key: {1} + Public Key: {2} + Secret Key: {3} + Sub Key: {4} + User IDs: {5} + """.format(dmsub.fpr, dmsub.primary, dmsub.pubkey, dmsub.seckey, dmsub.sub, + dmsub.uid)) + +As well as on the command line with: + +.. code:: shell + + bash-4.4$ gpg --homedir ~/.gnupg-dm -K + ~/.gnupg-dm/pubring.kbx + ---------------------- + sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] + 177B7C25DB99745EE2EE13ED026D2F19E99E63AA + uid [ultimate] Danger Mouse <[email protected]> + ssb rsa3072 2018-03-15 [E] [expires: 2018-09-13] + + bash-4.4$ + +.. _keygen-uids: + +User IDs +-------- + +.. _keygen-uids-add: + +Adding User IDs +~~~~~~~~~~~~~~~ + +By comparison to creating primary keys and subkeys, adding a new user ID +to an existing key is much simpler. The method used to do this is +``key_add_uid`` and the only arguments it takes are for the ``key`` and +the new ``uid``. + +.. code:: python + + import gpg + + c = gpg.Context() + c.home_dir = "~/.gnupg-dm" + + dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" + key = c.get_key(dmfpr, secret=True) + uid = "Danger Mouse <[email protected]>" + + c.key_add_uid(key, uid) + +Unsurprisingly the result of this is: + +.. code:: shell + + bash-4.4$ gpg --homedir ~/.gnupg-dm -K + ~/.gnupg-dm/pubring.kbx + ---------------------- + sec rsa3072 2018-03-15 [SC] [expires: 2019-03-15] + 177B7C25DB99745EE2EE13ED026D2F19E99E63AA + uid [ultimate] Danger Mouse <[email protected]> + uid [ultimate] Danger Mouse <[email protected]> + ssb rsa3072 2018-03-15 [E] [expires: 2018-09-13] + + bash-4.4$ + +.. _keygen-uids-revoke: + +Revokinging User IDs +~~~~~~~~~~~~~~~~~~~~ + +Revoking a user ID is a fairly similar process, except that it uses the +``key_revoke_uid`` method. + +.. code:: python + + import gpg + + c = gpg.Context() + c.home_dir = "~/.gnupg-dm" + + dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" + key = c.get_key(dmfpr, secret=True) + uid = "Danger Mouse <[email protected]>" + + c.key_revoke_uid(key, uid) + +.. _key-sign: + +Key certification +----------------- + +Since key certification is more frequently referred to as key signing, +the method used to perform this function is ``key_sign``. + +The ``key_sign`` method takes four arguments: ``key``, ``uids``, +``expires_in`` and ``local``. The default value of ``uids`` is ``None`` +and which results in all user IDs being selected. The default value of +both ``expires_in`` and ``local`` is ``False``; which results in the +signature never expiring and being able to be exported. + +The ``key`` is the key being signed rather than the key doing the +signing. To change the key doing the signing refer to the signing key +selection above for signing messages and files. + +If the ``uids`` value is not ``None`` then it must either be a string to +match a single user ID or a list of strings to match multiple user IDs. +In this case the matching of those strings must be precise and it is +case sensitive. + +To sign Danger Mouse\'s key for just the initial user ID with a +signature which will last a little over a month, do this: + +.. code:: python + + import gpg + + c = gpg.Context() + uid = "Danger Mouse <[email protected]>" + + dmfpr = "177B7C25DB99745EE2EE13ED026D2F19E99E63AA" + key = c.get_key(dmfpr, secret=True) + c.key_sign(key, uids=uid, expires_in=2764800) + +.. _advanced-use: + +Advanced or Experimental Use Cases +================================== + +.. _cython: + +C plus Python plus SWIG plus Cython +----------------------------------- + +In spite of the apparent incongruence of using Python bindings to a C +interface only to generate more C from the Python; it is in fact quite +possible to use the GPGME bindings with +`Cython <http://docs.cython.org/en/latest/index.html>`__. Though in many +cases the benefits may not be obvious since the most computationally +intensive work never leaves the level of the C code with which GPGME +itself is interacting with. + +Nevertheless, there are some situations where the benefits are +demonstrable. One of the better and easier examples being the one of the +early examples in this HOWTO, the `key +counting <#howto-keys-counting>`__ code. Running that example as an +executable Python script, ``keycount.py`` (available in the +``examples/howto/`` directory), will take a noticable amount of time to +run on most systems where the public keybox or keyring contains a few +thousand public keys. + +Earlier in the evening, prior to starting this section, I ran that +script on my laptop; as I tend to do periodically and timed it using +``time`` utility, with the following results: + +.. code:: shell + + bash-4.4$ time keycount.py + + Number of secret keys: 23 + Number of public keys: 12112 + + + real 11m52.945s + user 0m0.913s + sys 0m0.752s + + bash-4.4$ + +Sometime after that I imported another key and followed it with a little +test of Cython. This test was kept fairly basic, essentially lifting the +material from the `Cython Basic +Tutorial <http://docs.cython.org/en/latest/src/tutorial/cython_tutorial.html>`__ +to demonstrate compiling Python code to C. The first step was to take +the example key counting code quoted previously, essentially from the +importing of the ``gpg`` module to the end of the script: + +.. code:: python + + import gpg + + c = gpg.Context() + seckeys = c.keylist(pattern=None, secret=True) + pubkeys = c.keylist(pattern=None, secret=False) + + seclist = list(seckeys) + secnum = len(seclist) + + publist = list(pubkeys) + pubnum = len(publist) + + print(""" + Number of secret keys: {0} + Number of public keys: {1} + + """.format(secnum, pubnum)) + +Save that into a file called ``keycount.pyx`` and then create a +``setup.py`` file which contains this: + +.. code:: python + + from distutils.core import setup + from Cython.Build import cythonize + + setup( + ext_modules = cythonize("keycount.pyx") + ) + +Compile it: + +.. code:: shell + + bash-4.4$ python setup.py build_ext --inplace + bash-4.4$ + +Then run it in a similar manner to ``keycount.py``: + +.. code:: shell + + bash-4.4$ time python3.7 -c "import keycount" + + Number of secret keys: 23 + Number of public keys: 12113 + + + real 6m47.905s + user 0m0.785s + sys 0m0.331s + + bash-4.4$ + +Cython turned ``keycount.pyx`` into an 81KB ``keycount.o`` file in the +``build/`` directory, a 24KB ``keycount.cpython-37m-darwin.so`` file to +be imported into Python 3.7 and a 113KB ``keycount.c`` generated C +source code file of nearly three thousand lines. Quite a bit bigger than +the 314 bytes of the ``keycount.pyx`` file or the full 1,452 bytes of +the full executable ``keycount.py`` example script. + +On the other hand it ran in nearly half the time; taking 6 minutes and +47.905 seconds to run. As opposed to the 11 minutes and 52.945 seconds +which the CPython script alone took. + +The ``keycount.pyx`` and ``setup.py`` files used to generate this +example have been added to the ``examples/howto/advanced/cython/`` +directory The example versions include some additional options to +annotate the existing code and to detect Cython\'s use. The latter comes +from the `Magic +Attributes <http://docs.cython.org/en/latest/src/tutorial/pure.html#magic-attributes-within-the-pxd>`__ +section of the Cython documentation. + +.. _cheats-and-hacks: + +Miscellaneous extras and work-arounds +===================================== + +Most of the things in the following sections are here simply because +there was no better place to put them, even though some are only +peripherally related to the GPGME Python bindings. Some are also +workarounds for functions not integrated with GPGME as yet. This is +especially true of the first of these, dealing with `group +lines <#group-lines>`__. + +Group lines +----------- + +There is not yet an easy way to access groups configured in the gpg.conf +file from within GPGME. As a consequence these central groupings of keys +cannot be shared amongst multiple programs, such as MUAs readily. + +The following code, however, provides a work-around for obtaining this +information in Python. + +.. code:: python + + import subprocess + import sys + + if sys.platform == "win32": + gpgconfcmd = "gpgconf.exe --list-options gpg" + else: + gpgconfcmd = "gpgconf --list-options gpg" + + try: + lines = subprocess.getoutput(gpgconfcmd).splitlines() + except: + process = subprocess.Popen(gpgconfcmd.split(), stdout=subprocess.PIPE) + procom = process.communicate() + if sys.version_info[0] == 2: + lines = procom[0].splitlines() + else: + lines = procom[0].decode().splitlines() + + for i in range(len(lines)): + if lines[i].startswith("group") is True: + line = lines[i] + else: + pass + + groups = line.split(":")[-1].replace('"', '').split(',') + + group_lines = [] + group_lists = [] + + for i in range(len(groups)): + group_lines.append(groups[i].split("=")) + group_lists.append(groups[i].split("=")) + + for i in range(len(group_lists)): + group_lists[i][1] = group_lists[i][1].split() + +The result of that code is that ``group_lines`` is a list of lists where +``group_lines[i][0]`` is the name of the group and ``group_lines[i][1]`` +is the key IDs of the group as a string. + +The ``group_lists`` result is very similar in that it is a list of +lists. The first part, ``group_lists[i][0]`` matches +``group_lines[i][0]`` as the name of the group, but +``group_lists[i][1]`` is the key IDs of the group as a string. + +A demonstration of using the ``groups.py`` module is also available in +the form of the executable ``mutt-groups.py`` script. This second script +reads all the group entries in a user\'s ``gpg.conf`` file and converts +them into crypt-hooks suitable for use with the Mutt and Neomutt mail +clients. + +.. _hkp4py: + +Keyserver access for Python +--------------------------- + +The `hkp4py <https://github.com/Selfnet/hkp4py>`__ module by Marcel Fest +was originally a port of the old +`python-hkp <https://github.com/dgladkov/python-hkp>`__ module from +Python 2 to Python 3 and updated to use the +`requests <http://docs.python-requests.org/en/latest/index.html>`__ +module instead. It has since been modified to provide support for Python +2.7 as well and is available via PyPI. + +Since it rewrites the ``hkp`` protocol prefix as ``http`` and ``hkps`` +as ``https``, the module is able to be used even with servers which do +not support the full scope of keyserver functions. [6]_ It also works +quite readily when incorporated into a `Cython <#cython>`__ generated +and compiled version of any code. + +.. _hkp4py-strings: + +Key import format +~~~~~~~~~~~~~~~~~ + +The hkp4py module returns key data via requests as string literals +(``r.text``) instead of byte literals (``r.content``). This means that +the retrurned key data must be encoded to UTF-8 when importing that key +material using a ``gpg.Context().key_import()`` method. + +For this reason an alternative method has been added to the ``search`` +function of ``hkp4py.KeyServer()`` which returns the key in the correct +format as expected by ``key_import``. When importing using this module, +it is now possible to import with this: + +.. code:: python + + for key in keys: + if key.revoked is False: + gpg.Context().key_import(key.key_blob) + else: + pass + +Without that recent addition it would have been necessary to encode the +contents of each ``hkp4py.KeyServer().search()[i].key`` in +``hkp4py.KeyServer().search()`` before trying to import it. + +An example of this is included in the `Importing +Keys <#howto-import-key>`__ section of this HOWTO and the corresponding +executable version of that example is available in the +``lang/python/examples/howto`` directory as normal; the executable +version is the ``import-keys-hkp.py`` file. + +.. _copyright-and-license: + +Copyright and Licensing +======================= + +Copyright +--------- + +Copyright © The GnuPG Project, 2018. + +Copyright (C) The GnuPG Project, 2018. + +.. _draft-editions: + +Draft Editions of this HOWTO +---------------------------- + +Draft editions of this HOWTO may be periodically available directly from +the author at any of the following URLs: + +- `GPGME Python Bindings HOWTO draft (XHTML AWS S3 + SSL) <https://files.au.adversary.org/crypto/gpgme-python-howto.html>`__ +- `GPGME Python Bindings HOWTO draft (XHTML AWS S3 no + SSL) <http://files.au.adversary.org/crypto/gpgme-python-howto.html>`__ +- `GPGME Python Bindings HOWTO draft (Texinfo file AWS S3 + SSL) <https://files.au.adversary.org/crypto/gpgme-python-howto.texi>`__ +- `GPGME Python Bindings HOWTO draft (Texinfo file AWS S3 no + SSL) <http://files.au.adversary.org/crypto/gpgme-python-howto.texi>`__ +- `GPGME Python Bindings HOWTO draft (Info file AWS S3 + SSL) <https://files.au.adversary.org/crypto/gpgme-python-howto.info>`__ +- `GPGME Python Bindings HOWTO draft (Info file AWS S3 no + SSL) <http://files.au.adversary.org/crypto/gpgme-python-howto.info>`__ +- `GPGME Python Bindings HOWTO draft (reST file AWS S3 + SSL) <https://files.au.adversary.org/crypto/gpgme-python-howto.rst>`__ +- `GPGME Python Bindings HOWTO draft (reST file AWS S3 no + SSL) <http://files.au.adversary.org/crypto/gpgme-python-howto.rst>`__ +- `GPGME Python Bindings HOWTO draft (Docbook 4.2 AWS S3 + SSL) <https://files.au.adversary.org/crypto/gpgme-python-howto.xml>`__ +- `GPGME Python Bindings HOWTO draft (Docbook 4.2 AWS S3 no + SSL) <http://files.au.adversary.org/crypto/gpgme-python-howto.xml>`__ + +All of these draft versions except for one have been generated from this +document via Emacs `Org mode <https://orgmode.org/>`__ and `GNU +Texinfo <https://www.gnu.org/software/texinfo/>`__. Though it is likely +that the specific +`file <https://files.au.adversary.org/crypto/gpgme-python-howto.org>`__ +`version <http://files.au.adversary.org/crypto/gpgme-python-howto.org>`__ +used will be on the same server with the generated output formats. + +The one exception is the reStructuredText version, which was converted +using the latest version of Pandoc from the Org mode source file using +the following command: + +.. code:: shell + + pandoc -f org -t rst+smart -o gpgme-python-howto.rst gpgme-python-howto.org + +In addition to these there is a significantly less frequently updated +version as a HTML `WebHelp +site <https://files.au.adversary.org/crypto/gpgme-python-howto/webhelp/index.html>`__ +(AWS S3 SSL); generated from DITA XML source files, which can be found +in `an alternative +branch <https://dev.gnupg.org/source/gpgme/browse/ben%252Fhowto-dita/>`__ +of the GPGME git repository. + +These draft editions are not official documents and the version of +documentation in the master branch or which ships with released versions +is the only official documentation. Nevertheless, these draft editions +may occasionally be of use by providing more accessible web versions +which are updated between releases. They are provided on the +understanding that they may contain errors or may contain content +subject to change prior to an official release. + +.. _license: + +License GPL compatible +---------------------- + +This file is free software; as a special exception the author gives +unlimited permission to copy and/or distribute it, with or without +modifications, as long as this notice is preserved. + +This file is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY, to the extent permitted by law; without even the implied +warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + +Footnotes +========= + +.. [1] + ``short-history.org`` and/or ``short-history.html``. + +.. [2] + The ``lang/python/docs/`` directory in the GPGME source. + +.. [3] + With no issues reported specific to Python 3.7, the release of Python + 3.7.1 at around the same time as GPGME 1.12.0 and the testing with + Python 3.7.1rc1, there is no reason to delay moving 3.7 ahead of 3.6 + now. Production environments with more conservative requirements will + always enforce their own policies anyway and installation to each + supported minor release is quite possible too. + +.. [4] + Yes, even if you use virtualenv with everything you do in Python. If + you want to install this module as just your user account then you + will need to manually configure, compile and install the *entire* + GnuPG stack as that user as well. This includes libraries which are + not often installed that way. It can be done and there are + circumstances under which it is worthwhile, but generally only on + POSIX systems which utilise single user mode (some even require it). + +.. [5] + You probably don\'t really want to do this. Searching the keyservers + for \"gnupg.org\" produces over 400 results, the majority of which + aren\'t actually at the gnupg.org domain, but just included a comment + regarding the project in their key somewhere. + +.. [6] + Such as with ProtonMail servers. This also means that restricted + servers which only advertise either HTTP or HTTPS end points and not + HKP or HKPS end points must still be identified as as HKP or HKPS + within the Python Code. The ``hkp4py`` module will rewrite these + appropriately when the connection is made to the server. |