gpgme/lang/python/docs/GPGMEpythonHOWTOen.org

GNU Privacy Guard (GnuPG) Made Easy Python Bindings HOWTO (English)

• Introduction
  • Python 2 versus Python 3
• GPGME Concepts
  • A C API
  • Python bindings
  • Difference between the Python bindings and other GnuPG Python packages
    • The python-gnupg package maintained by Vinay Sajip
    • The gnupg package created and maintained by Isis Lovecruft
    • The PyME package maintained by Martin Albrecht
• GPGME Python bindings installation
  • No PyPI
  • Requirements
  • Installation
    • Installing GPGME
• Fundamentals
  • No REST
  • Context
• Working with keys
  • Key selection
  • Counting keys
• Basic Functions
  • Encryption
    • Encrypting to one key
    • Encrypting to multiple keys
  • Decryption
  • Signing text and files
    • Signing key selection
    • Normal or default signing messages or files
    • Detached signing messages and files
    • Clearsigning messages or text
  • Signature verification
• Miscellaneous work-arounds
  • Group lines
• Copyright and Licensing
  • Copyright (C) The GnuPG Project, 2018
  • License GPL compatible
• Footnotes

Introduction

Version:	0.0.1-alpha
Author:	Ben McGinnes <ben@gnupg.org>
Author GPG Key:	DB4724E6FA4286C92B4E55C4321E4E2373590E5D
Language:	English

This document provides basic instruction in how to use the GPGME Python bindings to programmatically leverage the GPGME library.

Python 2 versus Python 3

Though the GPGME Python bindings themselves provide support for both Python 2 and 3, the focus is unequivocally on Python 3 and specifically from Python 3.4 and above. As a consequence all the examples and instructions in this guide use Python 3 code.

Much of it will work with Python 2, but much of it also deals with Python 3 byte literals, particularly when reading and writing data. Developers concentrating on Python 2.7, and possibly even 2.6, will need to make the approprate modifications to support the older string and unicode types as opposted to bytes.

There are multiple reasons for concentrating on Python 3; some of which relate to the immediate integration of these bindings, some of which relate to longer term plans for both GPGME and the python bindings and some of which relate to the impending EOL period for Python 2.7. Essentially, though, there is little value in tying the bindings to a version of the language which is a dead end and the advantages offered by Python 3 over Python 2 make handling the data types with which GPGME deals considerably easier.

GPGME Concepts

A C API

Unlike many modern APIs with which programmers will be more familiar with these days, the GPGME API is a C API. The API is intended for use by C coders who would be able to access its features by including the gpgme.h header file eith their own C source code and then access its functions just as they would any other C headers.

This is a very effective method of gaining complete access to the API and in the most efficient manner possible. It does, however, have the drawback that it cannot be directly used by other languages without some means of providing an interface to those languages. This is where the need for bindings in various languages stems.

Python bindings

The Python bindings for GPGME provide a higher level means of accessing the complete feature set of GPGME itself. It also provides a more pythonic means of calling these API functions.

The bindings are generated dynamically with SWIG and the copy of gpgme.h generated when GPGME is compiled.

This means that a version of the Python bindings is fundamentally tied to the exact same version of GPGME used to generate that copy of gpgme.h.

Difference between the Python bindings and other GnuPG Python packages

There have been numerous attempts to add GnuPG support to Python over the years. Some of the most well known are listed here, along with what differentiates them.

The python-gnupg package maintained by Vinay Sajip

This is arguably the most popular means of integrating GPG with Python. The package utilises the subprocess module to implement wrappers for the gpg and gpg2 executables normally invoked on the command line (gpg.exe and gpg2.exe on Windows).

The popularity of this package stemmed from its ease of use and capability in providing the most commonly required features.

Unfortunately it has been beset by a number of security issues, most of which stemmed from using unsafe methods of accessing the command line via the subprocess calls.

The python-gnupg package is available under the MIT license.

The gnupg package created and maintained by Isis Lovecruft

In 2015 Isis Lovecruft from the Tor Project forked and then re-implemented the python-gnupg package as just gnupg. This new package also relied on subprocess to call the gpg or gpg2 binaries, but did so somewhat more securely.

However the naming and version numbering selected for this package resulted in conflicts with the original python-gnupg and since its functions were called in a different manner, the release of this package also resulted in a great deal of consternation when people installed what they thought was an upgrade that subsequently broke the code relying on it.

The gnupg package is available under the GNU General Public License version 3.0 (or later).

The PyME package maintained by Martin Albrecht

This package is the origin of these bindings, though they are somewhat different now. For details of when and how the PyME package was folded back into GPGME itself see the Short History document¹ in this Python bindings docs directory.²

The PyME package was first released in 2002 and was also the first attempt to implement a low level binding to GPGME. In doing so it provided access to considerably more functionality than either the python-gnupg or gnupg packages.

The PyME package is only available for Python 2.6 and 2.7.

Porting the PyME package to Python 3.4 in 2015 is what resulted in it being folded into the GPGME project and the current bindings are the end result of that effort.

The PyME package is available under the same dual licensing as GPGME itself: the GNU General Public License version 2.0 (or any later version) and the GNU Lesser Public License version 2.1 (or any later version).

GPGME Python bindings installation

No PyPI

Most third-party Python packages and modules are available and distributed through the Python Package Installer, known as PyPI.

Due to the nature of what these bindings are and how they work, it is infeasible to install the GPGME Python bindings in the same way.

Requirements

The GPGME Python bindings only have three requirements:

1. A suitable version of Python 2 or Python 3. With Python 2 that means Python 2.7 and with Python 3 that means Python 3.4 or higher.
2. SWIG.
3. GPGME itself. Which also means that all of GPGME's dependencies must be installed too.

Installation

Installing the Python bindings is effectively achieved by compiling and installing GPGME itself.

Once SWIG is installed with Python and all the dependencies for GPGME are installed you only need to confirm that the version(s) of Python you want the bindings installed for are in your $PATH.

By default GPGME will attempt to install the bindings for the most recent or highest version number of Python 2 and Python 3 it detects in $PATH. It specifically checks for the python and python3 executabled first and then checks for specific version numbers.

For Python 2 it checks for these executables in this order: python, python2 and python2.7.

For Python 3 it checks for these executables in this order: python3, python3.6, python3.5 and python3.4.

Installing GPGME

See the GPGME README file for details of how to install GPGME from source.

Fundamentals

Before we can get to the fun stuff, there are a few matters regarding GPGME's design which hold true whether you're dealing with the C code directly or these Python bindings.

No REST

The first part of which is or will be fairly blatantly obvious upon viewing the first example, but it's worth reiterating anyway. That being that this API is not a REST API. Nor indeed could it ever be one.

Most, if not all, Python programmers (and not just Python programmers) know how easy it is to work with a RESTful API. In fact they've become so popular that many other APIs attempt to emulate REST-like behaviour as much as they are able. Right down to the use of JSON formatted output to facilitate the use of their API without having to retrain developers.

This API does not do that. It would not be able to do that and also provide access to the entire C API on which it's built. It does, however, provide a very pythonic interface on top of the direct bindings and it's this pythonic layer with which this HOWTO deals with.

Context

One of the reasons which prevents this API from being RESTful is that most operations require more than one instruction to the API to perform the task. Sure, there are certain functions which can be performed simultaneously, particularly if the result known or strongly anticipated (e.g selecting and encrypting to a key known to be in the public keybox).

There are many more, however, which cannot be manipulated so readily: they must be performed in a specific sequence and the result of one operation has a direct bearing on the outcome of subsequent operations. Not merely by generating an error either.

When dealing with this type of persistant state on the web, full of both the RESTful and REST-like, it's most commonly referred to as a session. In GPGME, however, it is called a context and every operation type has one.

Working with keys

Key selection

Selecting keys to encrypt to or to sign with will be a common occurrence when working with GPGMe and the means available for doing so are quite simple.

They do depend on utilising a Context; however once the data is recorded in another variable, that Context does not need to be the same one which subsequent operations are performed.

The easiest way to select a specific key is by searching for that key's key ID or fingerprint, preferably the full fingerprint without any spaces in it. A long key ID will probably be okay, but is not advised and short key IDs are already a problem with some being generated to match specific patterns. It does not matter whether the pattern is upper or lower case.

So this is the best method:

  import gpg

  k = gpg.Context().keylist(pattern="258E88DCBD3CD44D8E7AB43F6ECB6AF0DEADBEEF")
  keys = list(k)

This is passable and very likely to be common:

  import gpg

  k = gpg.Context().keylist(pattern="0x6ECB6AF0DEADBEEF")
  keys = list(k)

And this is a really bad idea:

  import gpg

  k = gpg.Context().keylist(pattern="0xDEADBEEF")
  keys = list(k)

Alternatively it may be that the intention is to create a list of keys which all match a particular search string. For instance all the addresses at a particular domain, like this:

  import gpg

  ncsc = gpg.Context().keylist(pattern="ncsc.mil")
  nsa = list(ncsc)

Counting keys

Counting the number of keys in your public keybox (pubring.kbx), the format which has superceded the old keyring format (pubring.gpg and secring.gpg), or the number of secret keys is a very simple task.

  import gpg

  c = gpg.Context()
  seckeys = c.keylist(pattern=None, secret=True)
  pubkeys = c.keylist(pattern=None, secret=False)

  seclist = list(seckeys)
  secnum = len(seclist)

  publist = list(pubkeys)
  pubnum = len(publist)

  print("""
  Number of secret keys:  {0}
  Number of public keys:  {1}
  """.format(secnum, pubnum)

Basic Functions

The most frequently called features of any cryptographic library will be the most fundamental tasks for enxryption software. In this section we will look at how to programmatically encrypt data, decrypt it, sign it and verify signatures.

Encryption

Encrypting is very straight forward. In the first example below the message, text, is encrypted to a single recipient's key. In the second example the message will be encrypted to multiple recipients.

Encrypting to one key

The text is then encapsulated in a GPGME Data object as plain and the cipher object is created with another Data object. Then we create the Context as c and set it to use the ASCII armoured OpenPGP format. In later examples there will be alternative methods of setting the OpenPGP output to be ASCII armoured.

Next we prepare a keylist object in our Context and follow it with specifying the recipients as r. Note that the configuration in one's gpg.conf file is honoured, so if you have the options set to encrypt to one key or to a default key, that will be included with this operation.

This is followed by a quick check to be sure that the recipient is actually selected and that the key is available. Assuming it is, the encryption can proceed, but if not a message will print stating the key was not found.

The encryption operation is invoked within the Context with the c.op_encrypt function, loading the recipien (r), the message (plain) and the cipher. The cipher.seek uses os.SEEK_SET to set the data to the correct byte format for GPGME to use it.

At this point we no longer need the plaintext material, so we delete both the text and the plain objects. Then we write the encrypted data out to a file, secret_plans.txt.asc.

  import gpg
  import os

  rkey = "0x12345678DEADBEEF"
  text = """
  Some plain text to test with.  Obtained from any input source Python can read.

  It makes no difference whether it is string or bytes, but the bindings always
  produce byte output data.  Which is useful to know when writing out either the
  encrypted or decrypted results.

  """

  plain = gpg.core.Data(text)
  cipher = gpg.core.Data()
  c = gpg.core.Context()
  c.set_armor(1)

  c.op_keylist_start(rkey, 0)
  r = c.op_keylist_next()

  if r == None:
print("""The key for user "{0}" was not found""".format(rkey))
  else:
try:
   c.op_encrypt([r], 1, plain, cipher)
   cipher.seek(0, os.SEEK_SET)
   del(text)
   del(plain)
   afile = open("secret_plans.txt.asc", "wb")
   afile.write(cipher.read())
   afile.close()
except gpg.errors.GPGMEError as ex:
   print(ex.getstring())

Encrypting to multiple keys

Encrypting to multiple keys, in addition to a default key or a key configured to always encrypt to, is a little different and uses a slightly different call to the op_encrypt call demonstrated in the previous section.

The following example encrypts a message (text) to everyone with an email address on the gnupg.org domain,³ but does not encrypt to a default key or other key which is configured to normally encrypt to.

  import gpg

  text = b"""Oh look, another test message.

  The same rules apply as with the previous example and more likely
  than not, the message will actually be drawn from reading the
  contents of a file or, maybe, from entering data at an input()
  prompt.

  Since the text in this case must be bytes, it is most likely that
  the input form will be a separate file which is opened with "rb"
  as this is the simplest method of obtaining the correct data
  format.
  """

  c = gpg.Context(armor=True)
  rpattern = list(c.keylist(pattern="@gnupg.org", secret=False))
  logrus = []

  for i in range(len(rpattern)):
if rpattern[i].can_encrypt == 1:
   logrus.append(rpattern[i])

  cipher = c.encrypt(text, recipients=logrus, sign=False, always_trust=True)

  afile = open("secret_plans.txt.asc", "wb")
  afile.write(cipher[0])
  afile.close()

All it would take to change the above example to sign the message and also encrypt the message to any configured default keys would be to change the c.encrypt line to this:

  cipher = c.encrypt(text, recipients=logrus, always_trust=True,
add_encrypt_to=True)

The only keyword arguments requiring modification are those for which the default values are changing. The default value of sign is True, the default of always_trust is False, the default of add_encrypt_to is False.

If always_trust is not set to True and any of the recipient keys are not trusted (e.g. not signed or locally signed) then the encryption will raise an error. It is possible to mitigate this somewhat with something more like this:

  import gpg

  afile = open("secret_plans.txt", "rb")
  text = afile.read()
  afile.close()

  c = gpg.Context(armor=True)
  rpattern = list(c.keylist(pattern="@gnupg.org", secret=False))
  logrus = []

  for i in range(len(rpattern)):
if rpattern[i].can_encrypt == 1:
   logrus.append(rpattern[i])

  try:
cipher = c.encrypt(text, recipients=logrus, add_encrypt_to=True)
  except gpg.errors.InvalidRecipients as e:
for i in range(len(e.recipients)):
   for n in range(len(logrus)):
if logrus[n].fpr == e.recipients[i].fpr:
    logrus.remove(logrus[n])
              else:
                  pass
try:
   cipher = c.encrypt(text, recipients=logrus, add_encrypt_to=True)
except:
   pass

  afile = open("secret_plans.txt.asc", "wb")
  afile.write(cipher[0])
  afile.close()

This will attempt to encrypt to all the keys searched for, then remove invalid recipients if it fails and try again.

Decryption

Decrypting something encrypted to a key in one's secret keyring is fairly straight forward.

In this example code, however, preconfiguring either gpg.Context() or gpg.core.Context() as c is unnecessary because there is no need to modify the Context prior to conducting the decryption and since the Context is only used once, setting it to c simply adds lines for no gain.

  import os.path
  import gpg

  if os.path.exists("/path/to/secret_plans.txt.asc") is True:
ciphertext = "/path/to/secret_plans.txt.asc"
  elif os.path.exists("/path/to/secret_plans.txt.gpg") is True:
ciphertext = "/path/to/secret_plans.txt.gpg"
  else:
ciphertext = None

  if ciphertext is not None:
afile = open(ciphertext, "rb")
plaintext = gpg.Context().decrypt(afile)
afile.close()
newfile = open("/path/to/secret_plans.txt", "wb")
newfile.write(plaintext[0])
newfile.close()
print(plaintext[0])
plaintext[1]
plaintext[2]
del(plaintext)
  else:
pass

The data available in plaintext in this example is the decrypted content as a byte object in plaintext[0], the recipient key IDs and algorithms in plaintext[1] and the results of verifying any signatures of the data in plaintext[0].

Signing text and files

The following sections demonstrate how to specify

Signing key selection

By default GPGME and the Python bindings will use the default key configured for the user invoking the GPGME API. If there is no default key specified and there is more than one secret key available it may be necessary to specify the key or keys with which to sign messages and files.

  import gpg

  logrus = input("Enter the email address or string to match signing keys to: ")
  hancock = gpg.Context().keylist(pattern=logrus, secret=True)
  sig_src = list(hancock)

The signing examples in the following sections include the explicitly designated signers parameter in two of the five examples; once where the resulting signature would be ASCII armoured and once where it would not be armoured.

While it would be possible to enter a key ID or fingerprint here to match a specific key, it is not possible to enter two fingerprints and match two keys since the patten expects a string, bytes or None and not a list. A string with two fingerprints won't match any single key.

Normal or default signing messages or files

  import gpg

  text = b"""Declaration of ... something.

  """

  c = gpg.Context(armor=True, signers=sig_src)
  signed = c.sign(text, mode=0)

  afile = open("/path/to/statement.txt.asc", "wb")
  for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i]))
  afile.close()

  import gpg

  tfile = open("/path/to/statement.txt", "rb")
  text = tfile.read()
  tfile.close()

  c = gpg.Context()
  signed = c.sign(text, mode=0)

  afile = open("/path/to/statement.txt.sig", "wb")
  afile.write(signed[0])
  afile.close()

Detached signing messages and files

Detached ASCII Armoured signing:

  import gpg

  text = b"""Declaration of ... something.

  """

  c = gpg.Context(armor=True)
  signed = c.sign(text, mode=1)

  afile = open("/path/to/statement.txt.asc", "wb")
  for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i]))
  afile.close()

Detached binary signing of a file.

  import gpg

  tfile = open("/path/to/statement.txt", "rb")
  text = tfile.read()
  tfile.close()

  c = gpg.Context(signers=sig_src)
  signed = c.sign(text, mode=1)

  afile = open("/path/to/statement.txt.sig", "wb")
  afile.write(signed[0])
  afile.close()

Clearsigning messages or text

  import gpg

  text = """Declaration of ... something.

  """

  c = gpg.Context()
  signed = c.sign(text, mode=2)

  afile = open("/path/to/statement.txt.asc", "w")
  for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i].decode('utf-8')))
  afile.close()

Signature verification

Verify a signed file, both detached and not:

  import gpg
  import sys
  import time

  c = gpg.Context()

  data, result = c.verify(open(filename),
     open(detached_sig_filename)
     if detached_sig_filename else None)

  for index, sign in enumerate(result.signatures):
print("signature", index, ":")
print("  summary:     %#0x" % (sign.summary))
print("  status:      %#0x" % (sign.status))
print("  timestamp:  ", sign.timestamp)
print("  timestamp:  ", time.ctime(sign.timestamp))
print("  fingerprint:", sign.fpr)
print("  uid:        ", c.get_key(sign.fpr).uids[0].uid)

  if data:
sys.stdout.buffer.write(data)

Miscellaneous work-arounds

Group lines

There is not yet an easy way to access groups configured in the gpg.conf file from within GPGME. As a consequence these central groupings of keys cannot be shared amongst multiple programs, such as MUAs readily.

The following code, however, provides a work-around for obtaining this information in Python.

  import subprocess

  lines = subprocess.getoutput("gpgconf --list-options gpg").splitlines()

  for i in range(len(lines)):
if lines[i].startswith("group") is True:
   line = lines[i]
else:
   pass

  groups = line.split(":")[-1].replace('"', '').split(',')

  group_lines = groups
  for i in range(len(group_lines)):
group_lines[i] = group_lines[i].split("=")

  group_lists = group_lines
  for i in range(len(group_lists)):
group_lists[i][1] = group_lists[i][1].split()

The result of that code is that group_lines is a list of lists where group_lines[i][0] is the name of the group and group_lines[i][1] is the key IDs of the group as a string.

The group_lists result is very similar in that it is a list of lists. The first part, group_lists[i][0] matches group_lines[i][0] as the name of the group, but group_lists[i][1] is the key IDs of the group as a string.

Copyright and Licensing

Copyright (C) The GnuPG Project, 2018

Copyright © The GnuPG Project, 2018.

License GPL compatible

This file is free software; as a special exception the author gives unlimited permission to copy and/or distribute it, with or without modifications, as long as this notice is preserved.

This file is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY, to the extent permitted by law; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Footnotes

---

¹

Short_History.org and/or Short_History.html.

²

The lang/python/docs/ directory in the GPGME source.

³

You probably don't really want to do this. Searching the keyservers for "gnupg.org" produces over 400 results, the majority of which aren't actually at the gnupg.org domain, but just included a comment regarding the project in their key somewhere.