doc: Move keyformat.txt to here.

--

1 files changed, 520 insertions, 0 deletions

diff --git a/doc/keyformat.txt b/doc/keyformat.txt
new file mode 100644
index 000000000..dadfed4eb
--- /dev/null
+++ b/doc/keyformat.txt
@@ -0,0 +1,520 @@
+keyformat.txt               emacs, please switch to -*- org -*- mode
+-------------
+
+
+Some notes on the format of the secret keys used with gpg-agent.
+
+* Location of keys
+
+The secret keys[1] are stored on a per file basis in a directory below
+the ~/.gnupg home directory.  This directory is named
+
+   private-keys-v1.d
+
+and should have permissions 700.
+
+The secret keys are stored in files with a name matching the
+hexadecimal representation of the keygrip[2] and suffixed with ".key".
+
+* Extended Private Key Format
+
+** Overview
+GnuPG 2.3+ uses a new format to store private keys that is both
+more flexible and easier to read and edit by human beings.  The new
+format stores name,value-pairs using the common mail and http header
+convention.  Example (here indented with two spaces):
+
+  Description: Key to sign all GnuPG released tarballs.
+    The key is actually stored on a smart card.
+  Use-for-ssh: yes
+  OpenSSH-cert: long base64 encoded string wrapped so that this
+    key file can be easily edited with a standard editor.
+  Token: D2760001240102000005000011730000 OPENPGP.1 -
+  Token: FF020001008A77C1 PIV.9C -
+  Key: (shadowed-private-key
+    (rsa
+    (n #00AA1AD2A55FD8C8FDE9E1941772D9CC903FA43B268CB1B5A1BAFDC900
+    2961D8AEA153424DC851EF13B83AC64FBE365C59DC1BD3E83017C90D4365B4
+    83E02859FC13DB5842A00E969480DB96CE6F7D1C03600392B8E08EF0C01FC7
+    19F9F9086B25AD39B4F1C2A2DF3E2BE317110CFFF21D4A11455508FE407997
+    601260816C8422297C0637BB291C3A079B9CB38A92CE9E551F80AA0EBF4F0E
+    72C3F250461E4D31F23A7087857FC8438324A013634563D34EFDDCBF2EA80D
+    F9662C9CCD4BEF2522D8BDFED24CEF78DC6B309317407EAC576D889F88ADA0
+    8C4FFB480981FB68C5C6CA27503381D41018E6CDC52AAAE46B166BDC10637A
+    E186A02BA2497FDC5D1221#)
+    (e #00010001#)
+    (shadowed t1-v1
+     (#D2760001240102000005000011730000# OPENPGP.1)
+    )))
+
+GnuPG 2.2 is also able to read and write keys using the new format
+However, it only makes use of some of the values.
+
+Keys in the extended format can be recognized by looking at the first
+byte of the file.  If it starts with a '(' it is a naked S-expression,
+otherwise it is a key in extended format.
+
+*** Names
+A name must start with a letter and end with a colon.  Valid
+characters are all ASCII letters, numbers and the hyphen.  Comparison
+of names is done case insensitively.  Names may be used several times
+to represent an array of values.  Note that the name "Key" is special
+in that it is mandatory and must occur only once.
+
+*** Values
+Values are UTF-8 encoded strings.  Values can be wrapped at any point,
+and continued in the next line indicated by leading whitespace.  A
+continuation line with one leading space does not introduce a blank so
+that the lines can be effectively concatenated.  A blank line as part
+of a continuation line encodes a newline.
+
+*** Comments
+Lines containing only whitespace, and lines starting with whitespace
+followed by '#' are considered to be comments and are ignored.
+
+** Well known names
+*** Description
+This is a human readable string describing the key.
+
+*** Key
+The name "Key" is special in that it is mandatory and must occur only
+once.  The associated value holds the actual S-expression with the
+cryptographic key.  The S-expression is formatted using the 'Advanced
+Format' (GCRYSEXP_FMT_ADVANCED) that avoids non-printable characters
+so that the file can be easily inspected and edited.  See section
+'Private Key Format' below for details.
+
+*** Created
+The UTC time the key was created in ISO compressed format
+(yyyymmddThhmmss).  This information can be used to re-create an
+OpenPGP key.
+
+*** Label
+This is a short human readable description for the key which can be
+used by the software to describe the key in a user interface.  For
+example as part of the description in a prompt for a PIN or
+passphrase.  It is often used instead of a comment element as present
+in the S-expression of the "Key" item.
+
+*** OpenSSH-cert
+This takes a base64 encoded string wrapped so that this
+key file can be easily edited with a standard editor.  Several of such
+items can be used.
+
+*** Token
+If such an item exists it overrides the info given by the "shadow"
+parameter in the S-expression.  Using this item makes it possible to
+describe a key which is stored on several tokens and also makes it
+easy to update this info using a standard editor.  The syntax is
+similar to the "shadow" parameter:
+
+- Serialnumber of the token.
+- Key reference from the token in full format (e.g. "OpenPGP.2").
+- An optional fixed length of the PIN or "-".
+- The human readable serial number of a card.  This is usually what is
+  printed on the actual card.  This value is taken directly from the
+  card but when asking to insert a card it is useful to have this
+  value available.  GnuPG takes care of creating and possibly updating
+  this entry.  This is percent-plus-escaped.
+
+
+*** Use-for-ssh
+If given and the value is "yes" or "1" the key is allowed for use by
+gpg-agent's ssh-agent implementation.  This is thus the same as
+putting the keygrip into the 'sshcontrol' file.  Only one such item
+should exist.  If another non-zero value between 1 and 99999 is used,
+this is taken to establish the order in which the keys are returned to
+ssh; lower numbers are returned first.  If a negative value is used
+this overrides currently active (inserted) cards and thus allows to
+prefer on-disk keys over inserted cards.  A value of -1 has the
+highest priority; values are capped at -999 and have a lower priority
+but still above the positive values, inserted cards or the order in
+sshcontrol.
+
+
+*** Use-for-p11
+If given and the value is "yes" or "1" the key is allowed for use by
+GnuPG's PKCS#11 interface (Scute).  Note that Scute needs to be
+configured to use this optimization.
+
+*** Remote-list
+Allow to list the key with the KEYINFO command from a remote machine
+via the extra socket.  A boolean value is expected; the default is
+"no".  Note that KEYINFO will anyway provide information if the
+keygrip is specified.
+
+*** Confirm
+If given and the value is "yes", a user will be asked confirmation by
+a dialog window when the key is about to be used for
+PKSIGN/PKAUTH/PKDECRYPT operation.  If the value is "restricted", it
+is only asked for the access through extra/browser socket.
+
+*** Prompt
+This field is for card key.  If given and the value is "yes"
+(default), a user will be prompted about insertion of the card by a
+dialog window when card is not available.  When the value is "no", a
+card operation is refused with GPG_ERR_UNUSABLE_SECKEY error.
+
+*** Backup-info
+This gives information for a backup of the key.  The following fields
+are space delimited:
+
+- Hexified keygrip (uppercase) to make it easy to identify the
+  filename.  When restoring software should make sure that the keygrip
+  matches the one derived from the "Key" field.
+- Backup time in as ISO string.
+- Name of the backup software.
+- Arbitrary information.
+
+* Private Key Format
+** Unprotected Private Key Format
+
+The content of the file is an S-Expression like the ones used with
+Libgcrypt.  Here is an example of an unprotected file:
+
+(private-key
+ (rsa
+  (n #00e0ce9..[some bytes not shown]..51#)
+  (e #010001#)
+  (d #046129F..[some bytes not shown]..81#)
+  (p #00e861b..[some bytes not shown]..f1#)
+  (q #00f7a7c..[some bytes not shown]..61#)
+  (u #304559a..[some bytes not shown]..9b#)
+ )
+ (created-at timestamp)
+ (uri http://foo.bar x-foo:whatever_you_want)
+ (comment whatever)
+)
+
+"comment", "created-at" and "uri" are optional.  "comment" is
+currently used to keep track of ssh key comments. "created-at" is used
+to keep track of the creation time stamp used with OpenPGP keys; it is
+optional but required for some operations to calculate the fingerprint
+of the key.  This timestamp should be a string with the number of
+seconds since Epoch or an ISO time string (yyyymmddThhmmss).
+
+** Protected Private Key Format
+
+A protected key is like this:
+
+(protected-private-key
+   (rsa
+    (n #00e0ce9..[some bytes not shown]..51#)
+    (e #010001#)
+    (protected mode (parms) encrypted_octet_string)
+    (protected-at <isotimestamp>)
+   )
+   (uri http://foo.bar x-foo:whatever_you_want)
+   (comment whatever)
+)
+
+
+In this scheme the encrypted_octet_string is encrypted according to
+the algorithm described after the keyword protected; most protection
+algorithms need some parameters, which are given in a list before the
+encrypted_octet_string.  The result of the decryption process is a
+list of the secret key parameters.  The protected-at expression is
+optional; the isotimestamp is 15 bytes long (e.g. "19610711T172000").
+
+The currently defined protection modes are:
+
+*** openpgp-s2k3-sha1-aes-cbc
+
+  This describes an algorithm using AES in CBC mode for
+  encryption, SHA-1 for integrity protection and the String to Key
+  algorithm 3 from OpenPGP (rfc4880).
+
+  Example:
+
+  (protected openpgp-s2k3-sha1-aes-cbc
+    ((sha1 16byte_salt no_of_iterations) 16byte_iv)
+    encrypted_octet_string
+  )
+
+  The encrypted_octet string should yield this S-Exp (in canonical
+  representation) after decryption:
+
+  (
+   (
+    (d #046129F..[some bytes not shown]..81#)
+    (p #00e861b..[some bytes not shown]..f1#)
+    (q #00f7a7c..[some bytes not shown]..61#)
+    (u #304559a..[some bytes not shown]..9b#)
+   )
+   (hash sha1 #...[hashvalue]...#)
+  )
+
+  For padding reasons, random bytes are appended to this list - they can
+  easily be stripped by looking for the end of the list.
+
+  The hash is calculated on the concatenation of the public key and
+  secret key parameter lists: i.e. it is required to hash the
+  concatenation of these 6 canonical encoded lists for RSA, including
+  the parenthesis, the algorithm keyword and (if used) the protected-at
+  list.
+
+  (rsa
+   (n #00e0ce9..[some bytes not shown]..51#)
+   (e #010001#)
+   (d #046129F..[some bytes not shown]..81#)
+   (p #00e861b..[some bytes not shown]..f1#)
+   (q #00f7a7c..[some bytes not shown]..61#)
+   (u #304559a..[some bytes not shown]..9b#)
+   (protected-at "18950523T000000")
+  )
+
+  After decryption the hash must be recalculated and compared against
+  the stored one - If they don't match the integrity of the key is not
+  given.
+
+*** openpgp-s2k3-ocb-aes
+
+  This describes an algorithm using AES-128 in OCB mode, a nonce
+  of 96 bit, a taglen of 128 bit, and the String to Key algorithm 3
+  from OpenPGP (rfc4880).
+
+  Example:
+
+  (protected openpgp-s2k3-ocb-aes
+    ((sha1 16byte_salt no_of_iterations) 12byte_nonce)
+    encrypted_octet_string
+  )
+
+  The encrypted_octet string should yield this S-Exp (in canonical
+  representation) after decryption:
+
+  (
+   (
+    (d #046129F..[some bytes not shown]..81#)
+    (p #00e861b..[some bytes not shown]..f1#)
+    (q #00f7a7c..[some bytes not shown]..61#)
+    (u #304559a..[some bytes not shown]..9b#)
+   )
+  )
+
+  For padding reasons, random bytes may be appended to this list -
+  they can easily be stripped by looking for the end of the list.
+
+  The associated data required for this protection mode is the list
+  forming the public key parameters.  For the above example this is
+  is this canonical encoded S-expression:
+
+  (rsa
+   (n #00e0ce9..[some bytes not shown]..51#)
+   (e #010001#)
+   (protected-at "18950523T000000")
+  )
+
+*** openpgp-native
+
+  This is a wrapper around the OpenPGP Private Key Transport format
+  which resembles the standard OpenPGP format and allows the use of an
+  existing key without re-encrypting to the default protection format.
+
+  Example:
+
+  (protected openpgp-native
+    (openpgp-private-key
+     (version V)
+     (algo PUBKEYALGO)
+     (skey _ P1 _ P2 _ P3 ... e PN)
+     (csum n)
+     (protection PROTTYPE PROTALGO IV S2KMODE S2KHASH S2KSALT S2KCOUNT)))
+
+  Note that the public key parameters in SKEY are duplicated and
+  should be identical to their copies in the standard parameter
+  elements.  Here is an example of an entire protected private key
+  using this format:
+
+  (protected-private-key
+   (rsa
+    (n #00e0ce9..[some bytes not shown]..51#)
+    (e #010001#)
+    (protected openpgp-native
+     (openpgp-private-key
+      (version 4)
+      (algo rsa)
+      (skey _ #00e0ce9..[some bytes not shown]..51#
+            _ #010001#
+            e #.........................#)
+      (protection sha1 aes #aabbccddeeff00112233445566778899#
+                  3 sha1 #2596f93e85f41e53# 3:190))))
+   (uri http://foo.bar x-foo:whatever_you_want)
+   (comment whatever))
+
+** Shadowed Private Key Format
+
+To keep track of keys stored on IC cards we use a third format for
+private keys which are called shadow keys as they are only a reference
+to keys stored on a token:
+
+(shadowed-private-key
+   (rsa
+    (n #00e0ce9..[some bytes not shown]..51#)
+    (e #010001#)
+    (shadowed protocol (info))
+   )
+   (uri http://foo.bar x-foo:whatever_you_want)
+   (comment whatever)
+)
+
+The currently used protocols are "t1-v1" (token info version 1) and
+"tpm2-v1" (TPM format key information).  The second list with the
+information has this layout for "t1-v1":
+
+(card_serial_number id_string_of_key fixed_pin_length)
+
+FIXED_PIN_LENGTH is optional.  It can be used to store the length of
+the PIN; a value of 0 indicates that this information is not
+available.  The rationale for this field is that some pinpad equipped
+readers don't allow passing a variable length PIN.
+
+This is the (info) layout for "tpm2-v1":
+
+(parent tpm_private_string tpm_public_string)
+
+Although this precise format is encapsulated inside the tpm2daemon
+itself and nothing in gpg ever uses this.
+
+More items may be added to the list.
+
+** OpenPGP Private Key Transfer Format
+
+This format is used to transfer keys between gpg and gpg-agent.
+
+(openpgp-private-key
+  (version V)
+  (algo PUBKEYALGO)
+  (curve CURVENAME)
+  (skey _ P1 _ P2 _ P3 ... e PN)
+  (csum n)
+  (protection PROTTYPE PROTALGO IV S2KMODE S2KHASH S2KSALT S2KCOUNT))
+
+
+ * V is the packet version number (3 or 4).
+ * PUBKEYALGO is a Libgcrypt algo name
+ * CURVENAME is the name of the curve - only used with ECC.
+ * P1 .. PN are the parameters; the public parameters are never encrypted
+   the secret key parameters are encrypted if the "protection" list is
+   given.  To make this more explicit each parameter is preceded by a
+   flag "_" for cleartext or "e" for encrypted text.
+ * CSUM is the deprecated 16 bit checksum as defined by OpenPGP.  This
+   is an optional element.
+ * If PROTTYPE is "sha1" the new style SHA1 checksum is used if it is "sum"
+   the old 16 bit checksum (above) is used and if it is "none" no
+   protection at all is used.
+ * PROTALGO is a Libgcrypt style cipher algorithm name
+ * IV is the initialization vector.
+ * S2KMODE is the value from RFC-4880.
+ * S2KHASH is a libgcrypt style hash algorithm identifier.
+ * S2KSALT is the 8 byte salt
+ * S2KCOUNT is the count value from RFC-4880.
+
+** Persistent Passphrase Format
+
+Note: That this has not yet been implemented.
+
+To allow persistent storage of cached passphrases we use a scheme
+similar to the private-key storage format.  This is a master
+passphrase format where each file may protect several secrets under
+one master passphrase.  It is possible to have several of those files
+each protected by a dedicated master passphrase.  Clear text keywords
+allow listing the available protected passphrases.
+
+The name of the files with these protected secrets have this form:
+pw-<string>.dat.  STRING may be an arbitrary string, as a default name
+for the passphrase storage the name "pw-default.dat" is suggested.
+
+
+(protected-shared-secret
+   ((desc descriptive_text)
+    (key [key_1] (keyword_1 keyword_2 keyword_n))
+    (key [key_2] (keyword_21 keyword_22 keyword_2n))
+    (key [key_n] (keyword_n1 keyword_n2 keyword_nn))
+    (protected mode (parms) encrypted_octet_string)
+    (protected-at <isotimestamp>)
+   )
+)
+
+After decryption the encrypted_octet_string yields this S-expression:
+
+(
+ (
+  (value key_1 value_1)
+  (value key_2 value_2)
+  (value key_n value_n)
+ )
+ (hash sha1 #...[hashvalue]...#)
+)
+
+The "descriptive_text" is displayed with the prompt to enter the
+unprotection passphrase.
+
+KEY_1 to KEY_N are unique identifiers for the shared secret, for
+example an URI.  In case this information should be kept confidential
+as well, they may not appear in the unprotected part; however they are
+mandatory in the encrypted_octet_string.  The list of keywords is
+optional.  The order of the "key" lists and the order of the "value"
+lists must match, that is the first "key"-list is associated with the
+first "value" list in the encrypted_octet_string.
+
+The protection mode etc. is identical to the protection mode as
+described for the private key format.
+
+list of the secret key parameters.  The protected-at expression is
+optional; the isotimestamp is 15 bytes long (e.g. "19610711T172000").
+
+The "hash" in the encrypted_octet_string is calculated on the
+concatenation of the key list and value lists: i.e it is required to
+hash the concatenation of all these lists, including the
+parenthesis and (if used) the protected-at list.
+
+Example:
+
+(protected-shared-secret
+   ((desc "List of system passphrases")
+    (key "uid-1002" ("Knuth" "Donald Ervin Knuth"))
+    (key "uid-1001" ("Dijkstra" "Edsger Wybe Dijkstra"))
+    (key)
+    (protected mode (parms) encrypted_octet_string)
+    (protected-at "20100915T111722")
+   )
+)
+
+with "encrypted_octet_string" decoding to:
+
+(
+ (
+  (value 4:1002 "signal flags at the lock")
+  (value 4:1001 "taocp")
+  (value 1:0    "premature optimization is the root of all evil")
+ )
+ (hash sha1 #0102030405060708091011121314151617181920#)
+)
+
+To compute the hash this S-expression (in canonical format) was
+hashed:
+
+   ((desc "List of system passphrases")
+    (key "uid-1002" ("Knuth" "Donald Ervin Knuth"))
+    (key "uid-1001" ("Dijkstra" "Edsger Wybe Dijkstra"))
+    (key)
+    (value 4:1002 "signal flags at the lock")
+    (value 4:1001 "taocp")
+    (value 1:0    "premature optimization is the root of all evil")
+    (protected-at "20100915T111722")
+   )
+
+* Notes
+
+[1] I usually use the terms private and secret key exchangeable but prefer the
+term secret key because it can be visually be better distinguished
+from the term public key.
+
+[2] The keygrip is a unique identifier for a key pair, it is
+independent of any protocol, so that the same key can be used with
+different protocols.  PKCS-15 calls this a subjectKeyHash; it can be
+calculated using Libgcrypt's gcry_pk_get_keygrip ().
+
+[3] Even when canonical representation are required we will show the
+S-expression here in a more readable representation.