DIP-263 Use User Data for Public Keys (#276)

Problem
=======
See #263 

Solution
========
- Moved old Public Key Announcement to "Migrated Announcements".
- Added Avro type PublicKey.
- Updated User Data section to define key agreement and assertion method
key types.

---------

Co-authored-by: Wes Biggs <wes.biggs@amplica.io>

.spellcheckerdict.txt

@@ -25,6 +25,7 @@ DSNP
 DSNP-compatible
 DSNP-referenced
 ECIES
+EdDSA
 [Ee]num(s)?
 Ethereum
 Extrinsics
@@ -67,7 +68,7 @@ Polkadot
 Poly1305
 pre-configured
 PRId([ABs])?
-publicKey
+[Pp]ublicKey
 repo
 RFC[1-9][0-9]*
 Ristretto
@@ -78,6 +79,7 @@ schemaless
 Schnorrkel
 SDK
 SHA-256
+SHA-512
 Stateful
 stringified
 subkey

pages/DSNP/Announcements.md

@@ -17,9 +17,11 @@ Each Announcement has an enumerated type for use when separating out a stream of
 | 4 | [Reaction](Types/Reaction.md) | a public visual reply to a Broadcast | no | no |
 | 5 | [Profile](Types/Profile.md) | a profile | YES | no |
 | 6 | [Update](Types/Update.md) | an update to content| YES | no |
-| 7 | [Public Key](Types/PublicKey.md) | a public key for secure communication | no | no |
+| 7 | ~~[Public Key](Types/PublicKey.md)~~<sup>b</sup> | ~~a public key for secure communication~~ | ~~no~~ | ~~no~~ |
 
-<sup>a</sup> Since DSNP version 1.2, social graph changes use [User Data](UserData.md) operations as described in the [Graph](Graph.md) section.
+<sup>a</sup> Since DSNP version 1.2, social graph changes use [User Data](UserData.md) operations as described in the [Graph](Graph.md) section. 
+
+<sup>b</sup> Since DSNP version 1.3, public keys use [User Data](UserData.md) operations.
 
 ## Announcement Validation
 

pages/DSNP/Operations.md

@@ -34,7 +34,7 @@ Compliant implementations may respond to error conditions either synchronously,
 | <a id="remove-control-key">Remove Control Key</a> | YES | User | Key | [Control Key Removal Record](Records.md#control-key-removal) |
 | <a id="publish-announcement">Publish Announcement</a> | no* | User OR Delegate | [Announcement](Announcements.md) | [Announcement Published Record](Records.md#announcement-published) |
 | <a id="publish-batch">Publish Batch</a> | no* | User OR Delegate | [Announcement Type](Announcements.md#announcement-types), [Batch Publication](BatchPublications.md) URL, Batch Publication Content Hash | [Batch Published Record](Records.md#batch-published) |
-| <a id="get-user-data">Get User Data</a> | no | Any | User's Identifier, Set of Requested [User Data Types](UserData.md#user-data-types) | Map of [User Data Types](UserData.md#user-data-types) to [Data Chunks](UserData.md#data-chunks) with optional [Key Identifiers](Types/PublicKey.md#keyid) |
+| <a id="get-user-data">Get User Data</a> | no | Any | User's Identifier, Set of Requested [User Data Types](UserData.md#user-data-types) | Map of [User Data Types](UserData.md#user-data-types) to [Data Chunks](UserData.md#data-chunks) with optional key identifiers of encryption keys for each chunk |
 | <a id="replace-user-data">Replace User Data</a> | no | User OR Delegate | User's Identifier, [Key Identifier](Types/PublicKey.md#keyid), Map of [User Data Types](UserData.md#user-data-types) to [Data Chunks](UserData.md#data-chunks) | [User Data Replaced Record](Records.md#user-data-replaced) |
 
 \* For each Announcement Type, an implementation may support one or both of these operations.

pages/DSNP/Types/PublicKey.md

@@ -1,5 +1,7 @@
 # Public Key Announcement
 
+<mark>_Since DSNP version 1.3, public keys use [User Data](../UserData.md) operations._</mark>
+
 A Public Key Announcement is a way to note a new cryptographic key that can be used in DSNP to secure and verify the authenticity of communications.
 
 The most recently published key (if one exists) for a given key type should be treated as the active key of that key type.

pages/DSNP/Types/PublicKeyUserData.md

@@ -0,0 +1,37 @@
+# Public Key
+
+Represents an encoding of a public key, one half of a cryptographic key pair.
+
+## Serialization
+
+PublicKey object serialization MUST conform to the following [Avro](https://avro.apache.org) schema:
+
+```
+{
+    "namespace": "org.dsnp",
+    "name": "PublicKey",
+    "type": "record",
+    "fields": [
+        {
+            "name": "publicKey",
+            "type": "bytes",
+            "doc": "Multicodec public key"
+        }
+    ]
+}
+```
+
+## Generation
+
+### publicKey
+
+- MUST be a public key of an allowed key type for the associated User Data type, encoded in `multicodec` format
+
+The byte encoding consists of a [multicodec](https://github.com/multiformats/multicodec/blob/master/table.csv) key identifier (as a varint) followed by the public key's binary data in the codec's described format.
+
+#### Allowed Key Types
+
+| User Data Type | Allowed Algorithms ([multicodec](https://github.com/multiformats/multicodec/blob/master/table.csv)) | Purpose |
+| --- | --- | --- |
+| `keyAgreementPublicKeys` | `x25519-pub` |  A Curve25519 public key that can be used in key exchange protocols to generate a shared secret |
+| `assertionMethodPublicKeys` | `ed25519-pub` | A public key for the EdDSA signature scheme using SHA-512 and Curve25519 that can be used to verify cryptographic signatures |

pages/DSNP/UserData.md

@@ -19,8 +19,11 @@ DSNP implementations MUST support the following User Data Types:
 | <a name="private-follows">`privateFollows`</a> | 1.2 | `curve25519xsalsa20poly1305` |  [`DEFLATE`](https://en.wikipedia.org/wiki/Deflate) | [GraphEdge](Types/GraphEdge.md) |
 | <a name="private-connections">`privateConnections`</a> | 1.2 | `curve25519xsalsa20poly1305` | [`DEFLATE`](https://en.wikipedia.org/wiki/Deflate) | [GraphEdge](Types/GraphEdge.md) |
 | <a name="private-connection-prids">`privateConnectionPRIds`</a> | 1.2 | NONE | NONE | [PRId](Types/PRId.md) |
+| <a name="key-agreement-public-keys">`keyAgreementPublicKeys`</a> | 1.3 | NONE | NONE | [PublicKey](Types/PublicKeyUserData.md) |
+| <a name="assertion-method-public-keys">`assertionMethodPublicKeys`</a> | 1.3 | NONE | NONE | [PublicKey](Types/PublicKeyUserData.md) |
 
 Data for each data type is initially formatted as a stream of Avro objects that should conform to the schema specified.
+A DSNP system MAY limit the number of objects allowed for a given user data type; if so, this MUST be documented.
 Avro file- and block-level information (including in-stream schema) is omitted.
 The Avro stream is then compressed and/or encrypted as specified.
 
@@ -29,7 +32,7 @@ In the specification of cryptographic operations below, relevant methods from th
 
 ## Data Chunks
 
-Because blockchain systems often have specific limits to the amount of data that can be included in a given transaction, operations on user data deal with the data in discrete chunks.
+Because consensus systems often have specific limits to the amount of data that can be included in a given transaction, operations on user data deal with the data in discrete chunks.
 As implementation strategies may vary, implementations MUST define their own maximum chunk size in bytes to be used in the operations described below.
 
 ## Entity Tags
@@ -46,8 +49,8 @@ The Replace User Data Operation takes the following parameters:
 
 * A DSNP User Id
   * Implementations MUST ensure that the principal invoking this Operation is this user, or a transparent chain of delegation from the user to the principal exists.
-* A [Key Identifier](Types/PublicKey.md#keyid) for the `keyAgreement` key pair used to encrypt any private data in the operation.
-  (If only unencrypted user data types are included, the key identifier is optional.)
+* The index of the `keyAgreementPublicKeys` key pair used to encrypt any private data in the operation.
+  (If only unencrypted user data types are included, the key index may be omitted.)
 * A map containing the set of data types to update as the keys, and tuples consisting of (1) the schema version used to encode the data type, and (2) a list where each element includes a data chunk and its associated entity tag, as the values.
 
 If the Operation is successful, any previous data associated with the user for each data type included in the input MUST be removed and replaced by the new data.
@@ -61,9 +64,9 @@ Data chunks should be generated for each included data type using the following
 3. For each chunk generated, the application should then:
     1. If the data type requires compression, apply the compression codec noted.
     1. If the data type requires encryption,
-        1. Retrieve the user's active (most recently announced) `keyAgreement` public key, U<sub>public</sub>.
-  The `keyId` in the announcement should match the key identifier provided for this Operation.
-  If no key exists, one should be created and published as an Announcement before invoking the Operation.
+        1. Retrieve the user's active `keyAgreementPublicKey` key, U<sub>public</sub>, and note its index.
+  If no key exists, one should be created and published as User Data before invoking the Operation.
+  By convention, the key with the highest index (the last object in the Avro stream) is the active key.
         1. Create a sealed box (a payload encrypted with a symmetric key derived from an ephemeral key pair, and accompanied by the ephemeral public key), as in the [libsodium](https://doc.libsodium.org/public-key_cryptography/sealed_boxes) function `crypto_box_seal`, using U<sub>public</sub>.
         1. Include the previous `etag` value for the chunk. If the chunk is new, `etag` should be set to `null`.
   If any chunks are to be deleted, they should be included in the input identified with the existing `etag` and a `null` value for the data.
@@ -151,13 +154,13 @@ The Get User Data Operation takes the following parameters:
   * Note: While _writing_ user data is reserved for the user and any delegates, anyone on the network can read any user's data (though it may be encrypted).
 * The User Data Types (by system name) that should be retrieved.
 
-The operation returns a mapping of User Data Type to data chunks, with each data chunk annotated with an entity tag and (optionally) a key identifier. (Note that this is the same general structure as the input data for [Replace User Data](#replace-user-data-operation), for each requested data type.
+The operation returns a mapping of User Data Type to data chunks, with each data chunk annotated with an entity tag and (optionally) a key index. (Note that this is the same general structure as the input data for [Replace User Data](#replace-user-data-operation), for each requested data type.
 If no chunks for a requested data type exist, an implementation MAY omit that data type from the response.
 
 To transform the data from the output to Avro binary records, a consumer should apply the following algorithm to each data type included:
 1. Determine the relevant encryption algorithm, compression codec, and object schema from the User Data Type and version noted.
 1. For each chunk,
-    1. If encryption is indicated, decrypt the chunk data using the user's secret key (identified using the key identifier) as in the [libsodium](https://doc.libsodium.org/public-key_cryptography/sealed_boxes) function `crypto_box_seal_open`.
+    1. If encryption is indicated, decrypt the chunk data using the user's secret key (identified using the key index) as in the [libsodium](https://doc.libsodium.org/public-key_cryptography/sealed_boxes) function `crypto_box_seal_open`.
     1. If compression is required, uncompress the chunk data using the specified codec.
     1. Deserialize the uncompressed data to logical records according to the Avro object schema.
     1. Retain the chunk's `etag` value if needed for any updates.

pages/Frequency/Overview.md

@@ -19,18 +19,19 @@ Official schemas may be found in [GitHub](https://github.com/LibertyDSNP/schemas
 
 <!-- These ids are duplicated here for quick reference. -->
 
-| Name | Schema Id Mainnet | Schema Id Testnet (Paseo) | Schema Id Testnet (Rococo) |
-| --- | --- | --- | --- |
-| [Tombstone](./Publishing.md) | 1 | 1 | 1 |
-| [Broadcast](./Publishing.md) | 2 | 2 | 2 |
-| [Reply](./Publishing.md) | 3 | 3 | 3 |
-| [Reaction](./Publishing.md)| 4 | 4 | 4 |
-| [Profile](./Publishing.md) | 6 | 6 | 5 |
-| [Update](./Publishing.md)| 5 | 5 | 6 |
-| [Public Key](./Publishing.md)| 7 | 7 | 18 |
-| [Public Follows](./UserData.md)| 8 | 8 | 13 |
-| [Private Follows](./UserData.md) | 9 | 9 | 14 |
-| [Private Connections](./UserData.md) | 10 | 10 | 15 |
+| Name | Schema Name | Schema Id Mainnet | Schema Id Testnet (Paseo) | Schema Id Testnet (Rococo) |
+| --- | --- | --- | --- | --- |
+| [Tombstone](./Publishing.md) | `dsnp.tombstone` | 1 | 1 | 1 |
+| [Broadcast](./Publishing.md) | `dsnp.broadcast` | 2 | 2 | 2 |
+| [Reply](./Publishing.md) | `dsnp.reply` | 3 | 3 | 3 |
+| [Reaction](./Publishing.md)| `dsnp.reaction` | 4 | 4 | 4 |
+| [Profile](./Publishing.md) | `dsnp.profile` | 6 | 6 | 5 |
+| [Update](./Publishing.md)| `dsnp.update` | 5 | 5 | 6 |
+| [Key Agreement Public Keys](./UserData.md)| `dsnp.public-key-key-agreement` | 7 | 7 | 18 |
+| [Public Follows](./UserData.md)| `dsnp.public-follows` | 8 | 8 | 13 |
+| [Private Follows](./UserData.md) | `dsnp.private-follows` | 9 | 9 | 14 |
+| [Private Connections](./UserData.md) | `dsnp.private-connections` | 10 | 10 | 15 |
+| [Assertion Method Public Keys](./UserData.md)| `dsnp.public-key-assertion-method` | TBD | 11 | 100 |
 
 <!--
 ### Obsolete
@@ -50,4 +51,4 @@ Official schemas may be found in [GitHub](https://github.com/LibertyDSNP/schemas
 
 | Last Update Date | Frequency Release | DSNP Version |
 | --- | --- | --- |
-| 2023-07-17 | 1.5.2+ | 1.2.0 |
+| 2024-04-12 | 1.10.0+ | pre-1.3.0 |

pages/Frequency/Publishing.md

@@ -1,8 +1,7 @@
 # Announcement Publishing
 
-On Frequency, [Announcements](../DSNP/Announcements.md) are mapped to Schemas which in turn publish Frequency Messages or Stateful Storage changes.
+On Frequency, [Announcements](../DSNP/Announcements.md) are mapped to Schemas which in turn publish Frequency Messages.
 Frequency Messages are either individual Announcements from a particular user, or a Batch Publication with a multitude of possible users.
-Frequency Stateful Storage is either direct Announcements from a particular user or [User Data](./UserData.md) changes.
 
 <!-- Links to https://frequency-chain.github.io/frequency should be updated with links to docs.frequency.xyz when able to be -->
 
@@ -16,7 +15,6 @@ Frequency Stateful Storage is either direct Announcements from a particular user
 | 4 | [Reaction](../DSNP/Types/Reaction.md) | Batched | Mainnet: 4<br />Testnet (Paseo): 4<br />Testnet (Rococo): 4 | [`Parquet`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.Parquet) | [`IPFS`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.IPFS) |
 | 5 | [Profile](../DSNP/Types/Profile.md) | Batched | Mainnet: 6<br />Testnet (Paseo): 6<br />Testnet (Rococo): 5 | [`Parquet`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.Parquet) | [`IPFS`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.IPFS) |
 | 6 | [Update](../DSNP/Types/Update.md) | Batched | Mainnet: 5<br />Testnet (Paseo): 5<br />Testnet (Rococo): 6 | [`Parquet`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.Parquet) | [`IPFS`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.IPFS) |
-| 7 | [Public Key](../DSNP/Types/PublicKey.md) | [Stateful](./UserData.md#announcements) | Mainnet: 7<br />Testnet (Paseo): 7<br />Testnet (Rococo): 18 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Itemized`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Itemized) |
 
 Source code for each schema is located in the [LibertyDSNP/schemas](https://github.com/LibertyDSNP/schemas) repository.
 

pages/Frequency/UserData.md

@@ -1,27 +1,23 @@
 # Frequency User Data
 
-On Frequency, User Data and select Announcements are mapped to Schemas which use [Stateful Storage](https://frequency-chain.github.io/frequency/pallet_stateful_storage/index.html) for storage and retrieval of the data.
+On Frequency, User Data is mapped to Schemas which use [Stateful Storage](https://frequency-chain.github.io/frequency/pallet_stateful_storage/index.html) for storage and retrieval of the data.
 
 ## User Data Sets
 
 <!-- Update ./Overview.md if a Schema Id is updated -->
 
 | User Data Set | Deployed Schema Ids | Frequency Model Type | Frequency Payload Location | Settings |
 | --- | --- | --- | --- | --- |
+| [Public Key (Key Agreement)](../DSNP/Types/PublicKeyUserData.md) | Mainnet: 7<br />Testnet (Paseo): 7<br />Testnet (Rococo): 7 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Itemized`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Itemized) | [Append Only](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.AppendOnly), [Signature Required](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.SignatureRequired) |
 | [Public Follows](../DSNP/Graph.md#public-follows) | Mainnet: 8<br />Testnet (Paseo): 8<br />Testnet (Rococo): 13 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Paginated`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Paginated) | None |
 | [Private Follows](../DSNP/Graph.md#private-follows) | Mainnet: 9<br />Testnet (Paseo): 9<br />Testnet (Rococo): 14 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Paginated`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Paginated) | None |
 | [Private Connections](../DSNP/Graph.md#private-connections) | Mainnet: 10<br />Testnet (Paseo): 10<br />Testnet (Rococo): 15 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Paginated`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Paginated) | None |
+| [Public Key (Assertion Method)](../DSNP/Types/PublicKeyUserData.md) | Mainnet: TBD <br />Testnet (Paseo): 11<br />Testnet (Rococo): 100 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Itemized`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Itemized) | [Append Only](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.AppendOnly), [Signature Required](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.SignatureRequired) |
 
-[Pseudonymous Relationship Identifiers](./../DSNP/Graph.md#pseudonymous-relationship-identifiers) (PRIds) are stored along side Private Connections in the same Stateful Storage page.
+[Pseudonymous Relationship Identifiers](./../DSNP/Graph.md#pseudonymous-relationship-identifiers) (PRIds) are stored alongside Private Connections in the same Stateful Storage page.
 
 Source code for each schema is located in the [LibertyDSNP/schemas](https://github.com/LibertyDSNP/schemas) repository.
 
-## Announcements
-
-| Announcement | Deployed Schema Ids | Frequency Model Type | Frequency Payload Location | Settings |
-| --- | --- | --- | --- | --- |
-| [Public Key](../DSNP/Types/PublicKey.md) | Mainnet: 7<br />Testnet (Paseo): 7<br />Testnet (Rococo): 7 | [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary) | [`Itemized`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Itemized) | [Append Only](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.AppendOnly), [Signature Required](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.SignatureRequired) |
-
 ## Read Operation Mapping
 
 Stateful data is retrieved via state queries (`pallet.stateQuery`) or RPC calls (`pallet.rpcCall()`).