gpgme/lang/python/docs/GPGMEpythonHOWTOen.org

879 lines
28 KiB
Org Mode
Raw Normal View History

#+TITLE: GNU Privacy Guard (GnuPG) Made Easy Python Bindings HOWTO (English)
#+LATEX_COMPILER: xelatex
#+LATEX_CLASS: article
#+LATEX_CLASS_OPTIONS: [12pt]
#+LATEX_HEADER: \usepackage{xltxtra}
#+LATEX_HEADER: \usepackage[margin=1in]{geometry}
#+LATEX_HEADER: \setmainfont[Ligatures={Common}]{Times New Roman}
#+LATEX_HEADER: \author{Ben McGinnes <ben@gnupg.org>}
* Introduction
:PROPERTIES:
:CUSTOM_ID: intro
:END:
| Version: | 0.0.1-alpha |
| Author: | Ben McGinnes <ben@gnupg.org> |
| Author GPG Key: | DB4724E6FA4286C92B4E55C4321E4E2373590E5D |
| Language: | Australian English, British English |
| xml:lang: | en-AU, en-GB, en |
This document provides basic instruction in how to use the GPGME
Python bindings to programmatically leverage the GPGME library.
** Python 2 versus Python 3
:PROPERTIES:
:CUSTOM_ID: py2-vs-py3
:END:
Though the GPGME Python bindings themselves provide support for
both Python 2 and 3, the focus is unequivocally on Python 3 and
specifically from Python 3.4 and above. As a consequence all the
examples and instructions in this guide use Python 3 code.
Much of it will work with Python 2, but much of it also deals with
Python 3 byte literals, particularly when reading and writing data.
Developers concentrating on Python 2.7, and possibly even 2.6, will
need to make the approprate modifications to support the older
string and unicode types as opposted to bytes.
There are multiple reasons for concentrating on Python 3; some of
which relate to the immediate integration of these bindings, some
of which relate to longer term plans for both GPGME and the python
bindings and some of which relate to the impending EOL period for
Python 2.7. Essentially, though, there is little value in tying
the bindings to a version of the language which is a dead end and
the advantages offered by Python 3 over Python 2 make handling the
data types with which GPGME deals considerably easier.
* GPGME Concepts
:PROPERTIES:
:CUSTOM_ID: gpgme-concepts
:END:
** A C API
:PROPERTIES:
:CUSTOM_ID: gpgme-c-api
:END:
Unlike many modern APIs with which programmers will be more
familiar with these days, the GPGME API is a C API. The API is
intended for use by C coders who would be able to access its
features by including the =gpgme.h= header file eith their own C
source code and then access its functions just as they would any
other C headers.
This is a very effective method of gaining complete access to the
API and in the most efficient manner possible. It does, however,
have the drawback that it cannot be directly used by other
languages without some means of providing an interface to those
languages. This is where the need for bindings in various
languages stems.
** Python bindings
:PROPERTIES:
:CUSTOM_ID: gpgme-python-bindings
:END:
The Python bindings for GPGME provide a higher level means of
accessing the complete feature set of GPGME itself. It also
provides a more pythonic means of calling these API functions.
The bindings are generated dynamically with SWIG and the copy of
=gpgme.h= generated when GPGME is compiled.
This means that a version of the Python bindings is fundamentally
tied to the exact same version of GPGME used to generate that copy
of =gpgme.h=.
** Difference between the Python bindings and other GnuPG Python packages
:PROPERTIES:
:CUSTOM_ID: gpgme-python-bindings-diffs
:END:
There have been numerous attempts to add GnuPG support to Python
over the years. Some of the most well known are listed here, along
with what differentiates them.
*** The python-gnupg package maintained by Vinay Sajip
:PROPERTIES:
:CUSTOM_ID: diffs-python-gnupg
:END:
This is arguably the most popular means of integrating GPG with
Python. The package utilises the =subprocess= module to implement
wrappers for the =gpg= and =gpg2= executables normally invoked on
the command line (=gpg.exe= and =gpg2.exe= on Windows).
The popularity of this package stemmed from its ease of use and
capability in providing the most commonly required features.
Unfortunately it has been beset by a number of security issues,
most of which stemmed from using unsafe methods of accessing the
command line via the =subprocess= calls.
The python-gnupg package is available under the MIT license.
*** The gnupg package created and maintained by Isis Lovecruft
:PROPERTIES:
:CUSTOM_ID: diffs-isis-gnupg
:END:
In 2015 Isis Lovecruft from the Tor Project forked and then
re-implemented the python-gnupg package as just gnupg. This new
package also relied on subprocess to call the =gpg= or =gpg2=
binaries, but did so somewhat more securely.
However the naming and version numbering selected for this package
resulted in conflicts with the original python-gnupg and since its
functions were called in a different manner, the release of this
package also resulted in a great deal of consternation when people
installed what they thought was an upgrade that subsequently broke
the code relying on it.
The gnupg package is available under the GNU General Public
License version 3.0 (or later).
*** The PyME package maintained by Martin Albrecht
:PROPERTIES:
:CUSTOM_ID: diffs-pyme
:END:
This package is the origin of these bindings, though they are
somewhat different now. For details of when and how the PyME
package was folded back into GPGME itself see the /Short History/
document[fn:1] in this Python bindings =docs= directory.[fn:2]
The PyME package was first released in 2002 and was also the first
attempt to implement a low level binding to GPGME. In doing so it
provided access to considerably more functionality than either the
=python-gnupg= or =gnupg= packages.
The PyME package is only available for Python 2.6 and 2.7.
Porting the PyME package to Python 3.4 in 2015 is what resulted in
it being folded into the GPGME project and the current bindings
are the end result of that effort.
The PyME package is available under the same dual licensing as
GPGME itself: the GNU General Public License version 2.0 (or any
later version) and the GNU Lesser Public License version 2.1 (or
any later version).
* GPGME Python bindings installation
:PROPERTIES:
:CUSTOM_ID: gpgme-python-install
:END:
** No PyPI
:PROPERTIES:
:CUSTOM_ID: do-not-use-pypi
:END:
Most third-party Python packages and modules are available and
distributed through the Python Package Installer, known as PyPI.
Due to the nature of what these bindings are and how they work, it
is infeasible to install the GPGME Python bindings in the same way.
** Requirements
:PROPERTIES:
:CUSTOM_ID: gpgme-python-requirements
:END:
The GPGME Python bindings only have three requirements:
1. A suitable version of Python 2 or Python 3. With Python 2 that
means Python 2.7 and with Python 3 that means Python 3.4 or
higher.
2. SWIG.
3. GPGME itself. Which also means that all of GPGME's dependencies
must be installed too.
** Installation
:PROPERTIES:
:CUSTOM_ID: installation
:END:
Installing the Python bindings is effectively achieved by compiling
and installing GPGME itself.
Once SWIG is installed with Python and all the dependencies for
GPGME are installed you only need to confirm that the version(s) of
Python you want the bindings installed for are in your =$PATH=.
By default GPGME will attempt to install the bindings for the most
recent or highest version number of Python 2 and Python 3 it
detects in =$PATH=. It specifically checks for the =python= and
=python3= executabled first and then checks for specific version
numbers.
For Python 2 it checks for these executables in this order:
=python=, =python2= and =python2.7=.
For Python 3 it checks for these executables in this order:
=python3=, =python3.6=, =python3.5= and =python3.4=.
*** Installing GPGME
:PROPERTIES:
:CUSTOM_ID: install-gpgme
:END:
See the GPGME =README= file for details of how to install GPGME from
source.
* Fundamentals
:PROPERTIES:
:CUSTOM_ID: howto-fund-a-mental
:END:
Before we can get to the fun stuff, there are a few matters
regarding GPGME's design which hold true whether you're dealing with
the C code directly or these Python bindings.
** No REST
:PROPERTIES:
:CUSTOM_ID: no-rest-for-the-wicked
:END:
The first part of which is or will be fairly blatantly obvious upon
viewing the first example, but it's worth reiterating anyway. That
being that this API is /*not*/ a REST API. Nor indeed could it
ever be one.
Most, if not all, Python programmers (and not just Python
programmers) know how easy it is to work with a RESTful API. In
fact they've become so popular that many other APIs attempt to
emulate REST-like behaviour as much as they are able. Right down
to the use of JSON formatted output to facilitate the use of their
API without having to retrain developers.
This API does not do that. It would not be able to do that and
also provide access to the entire C API on which it's built. It
does, however, provide a very pythonic interface on top of the
direct bindings and it's this pythonic layer with which this HOWTO
deals with.
** Context
:PROPERTIES:
:CUSTOM_ID: howto-get-context
:END:
One of the reasons which prevents this API from being RESTful is
that most operations require more than one instruction to the API
to perform the task. Sure, there are certain functions which can
be performed simultaneously, particularly if the result known or
strongly anticipated (e.g selecting and encrypting to a key known
to be in the public keybox).
There are many more, however, which cannot be manipulated so
readily: they must be performed in a specific sequence and the
result of one operation has a direct bearing on the outcome of
subsequent operations. Not merely by generating an error either.
When dealing with this type of persistant state on the web, full of
both the RESTful and REST-like, it's most commonly referred to as a
session. In GPGME, however, it is called a context and every
operation type has one.
* Working with keys
:PROPERTIES:
:CUSTOM_ID: howto-keys
:END:
** Key selection
:PROPERTIES:
:CUSTOM_ID: howto-keys-selection
:END:
Selecting keys to encrypt to or to sign with will be a common
occurrence when working with GPGMe and the means available for
doing so are quite simple.
They do depend on utilising a Context; however once the data is
recorded in another variable, that Context does not need to be the
same one which subsequent operations are performed.
The easiest way to select a specific key is by searching for that
key's key ID or fingerprint, preferably the full fingerprint
without any spaces in it. A long key ID will probably be okay, but
is not advised and short key IDs are already a problem with some
being generated to match specific patterns. It does not matter
whether the pattern is upper or lower case.
So this is the best method:
#+begin_src python
import gpg
k = gpg.Context().keylist(pattern="258E88DCBD3CD44D8E7AB43F6ECB6AF0DEADBEEF")
keys = list(k)
#+end_src
This is passable and very likely to be common:
#+begin_src python
import gpg
k = gpg.Context().keylist(pattern="0x6ECB6AF0DEADBEEF")
keys = list(k)
#+end_src
And this is a really bad idea:
#+begin_src python
import gpg
k = gpg.Context().keylist(pattern="0xDEADBEEF")
keys = list(k)
#+end_src
Alternatively it may be that the intention is to create a list of
keys which all match a particular search string. For instance all
the addresses at a particular domain, like this:
#+begin_src python
import gpg
ncsc = gpg.Context().keylist(pattern="ncsc.mil")
nsa = list(ncsc)
#+end_src
** Counting keys
:PROPERTIES:
:CUSTOM_ID: howto-keys-counting
:END:
Counting the number of keys in your public keybox (=pubring.kbx=),
the format which has superceded the old keyring format
(=pubring.gpg= and =secring.gpg=), or the number of secret keys is
a very simple task.
#+begin_src python
import gpg
c = gpg.Context()
seckeys = c.keylist(pattern=None, secret=True)
pubkeys = c.keylist(pattern=None, secret=False)
seclist = list(seckeys)
secnum = len(seclist)
publist = list(pubkeys)
pubnum = len(publist)
print("""
Number of secret keys: {0}
Number of public keys: {1}
""".format(secnum, pubnum)
#+end_src
* Basic Functions
:PROPERTIES:
:CUSTOM_ID: howto-the-basics
:END:
The most frequently called features of any cryptographic library
will be the most fundamental tasks for enxryption software. In this
section we will look at how to programmatically encrypt data,
decrypt it, sign it and verify signatures.
** Encryption
:PROPERTIES:
:CUSTOM_ID: howto-basic-encryption
:END:
Encrypting is very straight forward. In the first example below
the message, =text=, is encrypted to a single recipient's key. In
the second example the message will be encrypted to multiple
recipients.
*** Encrypting to one key
:PROPERTIES:
:CUSTOM_ID: howto-basic-encryption-single
:END:
The text is then encapsulated in a GPGME Data object as =plain= and
the =cipher= object is created with another Data object. Then we
create the Context as =c= and set it to use the ASCII armoured
OpenPGP format. In later examples there will be alternative
methods of setting the OpenPGP output to be ASCII armoured.
Next we prepare a keylist object in our Context and follow it with
specifying the recipients as =r=. Note that the configuration in
one's =gpg.conf= file is honoured, so if you have the options set
to encrypt to one key or to a default key, that will be included
with this operation.
This is followed by a quick check to be sure that the recipient is
actually selected and that the key is available. Assuming it is,
the encryption can proceed, but if not a message will print stating
the key was not found.
The encryption operation is invoked within the Context with the
=c.op_encrypt= function, loading the recipien (=r=), the message
(=plain=) and the =cipher=. The =cipher.seek= uses =os.SEEK_SET=
to set the data to the correct byte format for GPGME to use it.
At this point we no longer need the plaintext material, so we
delete both the =text= and the =plain= objects. Then we write the
encrypted data out to a file, =secret_plans.txt.asc=.
#+begin_src python
import gpg
import os
rkey = "0x12345678DEADBEEF"
text = """
Some plain text to test with. Obtained from any input source Python can read.
It makes no difference whether it is string or bytes, but the bindings always
produce byte output data. Which is useful to know when writing out either the
encrypted or decrypted results.
"""
plain = gpg.core.Data(text)
cipher = gpg.core.Data()
c = gpg.core.Context()
c.set_armor(1)
c.op_keylist_start(rkey, 0)
r = c.op_keylist_next()
if r == None:
print("""The key for user "{0}" was not found""".format(rkey))
else:
try:
c.op_encrypt([r], 1, plain, cipher)
cipher.seek(0, os.SEEK_SET)
del(text)
del(plain)
afile = open("secret_plans.txt.asc", "wb")
afile.write(cipher.read())
afile.close()
except gpg.errors.GPGMEError as ex:
print(ex.getstring())
#+end_src
*** Encrypting to multiple keys
:PROPERTIES:
:CUSTOM_ID: howto-basic-encryption-multiple
:END:
Encrypting to multiple keys, in addition to a default key or a key
configured to always encrypt to, is a little different and uses a
slightly different call to the =op_encrypt call= demonstrated in the
previous section.
The following example encrypts a message (=text=) to everyone with
an email address on the =gnupg.org= domain,[fn:3] but does /not/ encrypt
to a default key or other key which is configured to normally
encrypt to.
#+begin_src python
import gpg
text = b"""Oh look, another test message.
The same rules apply as with the previous example and more likely
than not, the message will actually be drawn from reading the
contents of a file or, maybe, from entering data at an input()
prompt.
Since the text in this case must be bytes, it is most likely that
the input form will be a separate file which is opened with "rb"
as this is the simplest method of obtaining the correct data
format.
"""
c = gpg.Context(armor=True)
rpattern = list(c.keylist(pattern="@gnupg.org", secret=False))
logrus = []
for i in range(len(rpattern)):
if rpattern[i].can_encrypt == 1:
logrus.append(rpattern[i])
cipher = c.encrypt(text, recipients=logrus, sign=False, always_trust=True)
afile = open("secret_plans.txt.asc", "wb")
afile.write(cipher[0])
afile.close()
#+end_src
All it would take to change the above example to sign the message
and also encrypt the message to any configured default keys would
be to change the =c.encrypt= line to this:
#+begin_src python
cipher = c.encrypt(text, recipients=logrus, always_trust=True,
add_encrypt_to=True)
#+end_src
The only keyword arguments requiring modification are those for
which the default values are changing. The default value of
=sign= is =True=, the default of =always_trust= is =False=, the
default of =add_encrypt_to= is =False=.
If =always_trust= is not set to =True= and any of the recipient
keys are not trusted (e.g. not signed or locally signed) then the
encryption will raise an error. It is possible to mitigate this
somewhat with something more like this:
#+begin_src python
import gpg
afile = open("secret_plans.txt", "rb")
text = afile.read()
afile.close()
c = gpg.Context(armor=True)
rpattern = list(c.keylist(pattern="@gnupg.org", secret=False))
logrus = []
for i in range(len(rpattern)):
if rpattern[i].can_encrypt == 1:
logrus.append(rpattern[i])
try:
cipher = c.encrypt(text, recipients=logrus, add_encrypt_to=True)
except gpg.errors.InvalidRecipients as e:
for i in range(len(e.recipients)):
for n in range(len(logrus)):
if logrus[n].fpr == e.recipients[i].fpr:
logrus.remove(logrus[n])
else:
pass
try:
cipher = c.encrypt(text, recipients=logrus, add_encrypt_to=True)
except:
pass
afile = open("secret_plans.txt.asc", "wb")
afile.write(cipher[0])
afile.close()
#+end_src
This will attempt to encrypt to all the keys searched for, then
remove invalid recipients if it fails and try again.
** Decryption
:PROPERTIES:
:CUSTOM_ID: howto-basic-encryption
:END:
Decrypting something encrypted to a key in one's secret keyring is
fairly straight forward.
In this example code, however, preconfiguring either
=gpg.Context()= or =gpg.core.Context()= as =c= is unnecessary
because there is no need to modify the Context prior to conducting
the decryption and since the Context is only used once, setting it
to =c= simply adds lines for no gain.
#+begin_src python
import os.path
import gpg
if os.path.exists("/path/to/secret_plans.txt.asc") is True:
ciphertext = "/path/to/secret_plans.txt.asc"
elif os.path.exists("/path/to/secret_plans.txt.gpg") is True:
ciphertext = "/path/to/secret_plans.txt.gpg"
else:
ciphertext = None
if ciphertext is not None:
afile = open(ciphertext, "rb")
plaintext = gpg.Context().decrypt(afile)
afile.close()
newfile = open("/path/to/secret_plans.txt", "wb")
newfile.write(plaintext[0])
newfile.close()
print(plaintext[0])
plaintext[1]
plaintext[2]
del(plaintext)
else:
pass
#+end_src
The data available in plaintext in this example is the decrypted
content as a byte object in =plaintext[0]=, the recipient key IDs
and algorithms in =plaintext[1]= and the results of verifying any
signatures of the data in =plaintext[0]=.
** Signing text and files
:PROPERTIES:
:CUSTOM_ID: howto-basic-signing
:END:
The following sections demonstrate how to specify
*** Signing key selection
:PROPERTIES:
:CUSTOM_ID: howto-basic-signing-signers
:END:
By default GPGME and the Python bindings will use the default key
configured for the user invoking the GPGME API. If there is no
default key specified and there is more than one secret key
available it may be necessary to specify the key or keys with
which to sign messages and files.
#+begin_src python
import gpg
logrus = input("Enter the email address or string to match signing keys to: ")
hancock = gpg.Context().keylist(pattern=logrus, secret=True)
sig_src = list(hancock)
#+end_src
The signing examples in the following sections include the
explicitly designated =signers= parameter in two of the five
examples; once where the resulting signature would be ASCII
armoured and once where it would not be armoured.
While it would be possible to enter a key ID or fingerprint here
to match a specific key, it is not possible to enter two
fingerprints and match two keys since the patten expects a string,
bytes or None and not a list. A string with two fingerprints
won't match any single key.
*** Normal or default signing messages or files
:PROPERTIES:
:CUSTOM_ID: howto-basic-signing-normal
:END:
The normal or default signing process is essentially the same as
is most often invoked when also encrypting a message or file. So
when the encryption component is not utilised, the result is to
produce an encoded and signed output which may or may not be ASCII
armoured and which may or may not also be compressed.
By default compression will be used unless GnuPG detects that the
plaintext is already compressed. ASCII armouring will be
determined according to the value of =gpg.Context().armor=.
#+begin_src python
import gpg
text = b"""Declaration of ... something.
"""
c = gpg.Context(armor=True, signers=sig_src)
signed = c.sign(text, mode=0)
afile = open("/path/to/statement.txt.asc", "wb")
for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i]))
afile.close()
#+end_src
#+begin_src python
import gpg
tfile = open("/path/to/statement.txt", "rb")
text = tfile.read()
tfile.close()
c = gpg.Context()
signed = c.sign(text, mode=0)
afile = open("/path/to/statement.txt.sig", "wb")
afile.write(signed[0])
afile.close()
#+end_src
*** Detached signing messages and files
:PROPERTIES:
:CUSTOM_ID: howto-basic-signing-detached
:END:
Detached ASCII Armoured signing:
#+begin_src python
import gpg
text = b"""Declaration of ... something.
"""
c = gpg.Context(armor=True)
signed = c.sign(text, mode=1)
afile = open("/path/to/statement.txt.asc", "wb")
for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i]))
afile.close()
#+end_src
Detached binary signing of a file.
#+begin_src python
import gpg
tfile = open("/path/to/statement.txt", "rb")
text = tfile.read()
tfile.close()
c = gpg.Context(signers=sig_src)
signed = c.sign(text, mode=1)
afile = open("/path/to/statement.txt.sig", "wb")
afile.write(signed[0])
afile.close()
#+end_src
*** Clearsigning messages or text
:PROPERTIES:
:CUSTOM_ID: howto-basic-signing-clear
:END:
#+begin_src python
import gpg
text = """Declaration of ... something.
"""
c = gpg.Context()
signed = c.sign(text, mode=2)
afile = open("/path/to/statement.txt.asc", "w")
for i in range(len(signed[0].splitlines())):
afile.write("{0}\n".format(signed[0].splitlines()[i].decode('utf-8')))
afile.close()
#+end_src
** Signature verification
:PROPERTIES:
:CUSTOM_ID: howto-basic-verification
:END:
Verify a signed file, both detached and not:
#+begin_src python
import gpg
import sys
import time
c = gpg.Context()
data, result = c.verify(open(filename),
open(detached_sig_filename)
if detached_sig_filename else None)
for index, sign in enumerate(result.signatures):
print("signature", index, ":")
print(" summary: %#0x" % (sign.summary))
print(" status: %#0x" % (sign.status))
print(" timestamp: ", sign.timestamp)
print(" timestamp: ", time.ctime(sign.timestamp))
print(" fingerprint:", sign.fpr)
print(" uid: ", c.get_key(sign.fpr).uids[0].uid)
if data:
sys.stdout.buffer.write(data)
#+end_src
* Miscellaneous work-arounds
:PROPERTIES:
:CUSTOM_ID: cheats-and-hacks
:END:
** Group lines
:PROPERTIES:
:CUSTOM_ID: group-lines
:END:
There is not yet an easy way to access groups configured in the
gpg.conf file from within GPGME. As a consequence these central
groupings of keys cannot be shared amongst multiple programs, such
as MUAs readily.
The following code, however, provides a work-around for obtaining
this information in Python.
#+begin_src python
import subprocess
lines = subprocess.getoutput("gpgconf --list-options gpg").splitlines()
for i in range(len(lines)):
if lines[i].startswith("group") is True:
line = lines[i]
else:
pass
groups = line.split(":")[-1].replace('"', '').split(',')
group_lines = groups
for i in range(len(group_lines)):
group_lines[i] = group_lines[i].split("=")
group_lists = group_lines
for i in range(len(group_lists)):
group_lists[i][1] = group_lists[i][1].split()
#+end_src
The result of that code is that =group_lines= is a list of lists
where =group_lines[i][0]= is the name of the group and
=group_lines[i][1]= is the key IDs of the group as a string.
The =group_lists= result is very similar in that it is a list of
lists. The first part, =group_lists[i][0]= matches
=group_lines[i][0]= as the name of the group, but
=group_lists[i][1]= is the key IDs of the group as a string.
* Copyright and Licensing
:PROPERTIES:
:CUSTOM_ID: copyright-and-license
:END:
** Copyright (C) The GnuPG Project, 2018
:PROPERTIES:
:CUSTOM_ID: copyright
:END:
Copyright © The GnuPG Project, 2018.
** License GPL compatible
:PROPERTIES:
:CUSTOM_ID: license
:END:
This file is free software; as a special exception the author gives
unlimited permission to copy and/or distribute it, with or without
modifications, as long as this notice is preserved.
This file is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY, to the extent permitted by law; without even
the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
* Footnotes
[fn:1] =Short_History.org= and/or =Short_History.html=.
[fn:2] The =lang/python/docs/= directory in the GPGME source.
[fn:3] You probably don't really want to do this. Searching the
keyservers for "gnupg.org" produces over 400 results, the majority of
which aren't actually at the gnupg.org domain, but just included a
comment regarding the project in their key somewhere.