DRIVERS-2926 BSON Binary Vector Subtype Support

source/bson-binary-vector/bson-binary-vector.md

@@ -0,0 +1,103 @@
+# BSON Binary Subtype 9 - Vector
+
+- Status: Pending
+- Minimum Server Version: N/A
+
+______________________________________________________________________
+
+## Abstract
+
+This document describes the addition of a new subtype to the Binary BSON type. This subtype is used for efficient
+storage and retrieval of vectors. Vectors here refer to densely packed arrays of numbers, all of the same type.
+
+## Motivation
+
+These representations correspond to the numeric types supported by popular numerical libraries for vector processing,
+such as NumPy, PyTorch, TensorFlow and Apache Arrow. Storing and retrieving vector data using the same densely packed
+format used by these libraries can result in up to significant memory savings and processing efficiency.
+
+### META
+
+The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
+"OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
+
+## Specification
+
+This specification introduces a new BSON binary subtype, the vector, with value `"\x09"`.
+
+Drivers SHOULD provide idiomatic APIs to translate between arrays of numbers and this BSON Binary specification.
+
+#### Data Types
+
+Each vector can take one of multiple data types (dtypes). The following table lists the dtypes implemented.
+
+| Vector data type | Alias      | Bits per vector element | [Arrow Data Type](https://arrow.apache.org/docs/cpp/api/datatype.html) (for illustration) |
+| ---------------- | ---------- | ----------------------- | ----------------------------------------------------------------------------------------- |
+| `0x03`           | INT8       | 8                       | INT8                                                                                      |
+| `0x27`           | FLOAT32    | 32                      | FLOAT                                                                                     |
+| `0x10`           | PACKED_BIT | 1     `*`               | BOOL                                                                                      |
+
+`*` A Binary Quantized (PACKED_BIT) Vector is a vector of 0s and 1s (bits), but it is represented in memory as a list of
+integers in \[0, 255\]. So, for example, the vector `[0, 255]` would be shorthand for the 16 bit vector
+`[0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1]`. The idea is that each number (a uint8) can be stored as a single byte. Of course,
+some languages, Python for one, do not have an uint8 type, so must be represented as an int in memory, but not on disk.
+
+The authors are well-aware of the inherent ambiguity here, and alternatives. This is a market-standard, unfortunately.
+Change is inevitable.
+
+#### Byte padding
+
+As not all data types have a bit length equal to a multiple of 8, and hence do not fit squarely into a certain number of
+bytes, a second piece of metadata, the "padding" is included. This instructs the driver of the number of bits in the
+final byte that are to be ignored.
+
+#### Binary structure
+
+Following the binary subtype `0x09` a two-element byte array of metadata precedes the packed numbers.
+
+- The first byte (dtype) describes its data type. The table above shows those that MUST be implemented. This table may
+  increase.
+
+- The second byte (padding) prescribes the number of bits to ignore in the final byte of the value.
+
+- The remainder contains the actual vector elements packed according to dtype.
+
+For example, a vector `[6, 7]` of dtype PACKED_BIT (`\x10`) with a padding of `3` would look like this:
+`b"\x10\x03\x06\x07'`: 1 byte for dtype, 1 for padding, and 1 for each uint8.
+
+<table border="1" cellspacing="0" cellpadding="5">
+  <tr>
+    <td colspan="8">1st byte: dtype (from list in previous table) </td>
+    <td colspan="8">2nd byte: padding (values in [0,7])</td>
+    <td colspan="1">binary numbers packed according to dtype</td>
+  </tr>
+  <tr>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>1</td>
+    <td>0</td>
+    <td>1</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>0</td>
+    <td>1</td>
+    <td>1</td>
+    <td>...</td>
+  </tr>
+</table>
+
+All values use the little-endian format.
+
+### Reference Implementation
+
+- PYTHON (PYTHON-4577) [pymongo.binary](https://github.com/mongodb/mongo-python-driver/blob/master/bson/binary.py)
+
+### Test Plan
+
+See the [README](tests/README.md) for tests.

source/bson-binary-vector/tests/README.md

@@ -0,0 +1,40 @@
+# Testing Binary subtype 9: Vector
+
+The JSON files in this directory tree are platform-independent tests that drivers can use to prove their conformance to
+the specification.
+
+These tests focus on the roundtrip of the list numbers as input/output, along with their data type and byte padding.
+
+Additional tests exist in `bson_corpus/tests/binary.json` but do not sufficiently test the end-to-end process of Vector
+to BSON. For this reason, drivers must create a bespoke test runner for the vector subtype.
+
+Each test case here pertains to a single vector. The inputs required to create the Binary BSON object are defined, and
+when valid, the Canonical BSON and Extended JSON representations are included for comparison.
+
+## Version
+
+Files in the "specifications" repository have no version scheme. They are not tied to a MongoDB server version.
+
+## Format
+
+#### Top level keys
+
+Each JSON file contains three top-level keys.
+
+- `description`: human-readable description of what is in the file
+- `test_key`: Field name used when decoding/encoding a BSON document containing the single BSON Binary for the test
+  case. Applies to *every* case.
+- `tests`: array of test case objects, each of which have the following keys. Valid cases will also contain additional
+  binary and json encoding values.
+
+#### Keys of tests objects
+
+- `description`: string describing the test.
+- `valid`: boolean indicating if the vector, dtype, and padding should be considered a valid input.
+- `vector`: list of numbers
+- `dtype_hex`: string defining the data type in hex (e.g. "0x10", "0x27")
+- `dtype_alias`: (optional) string defining the data dtype, perhaps as Enum.
+- `padding`: (optional) integer for byte padding. Defaults to 0.
+- `canonical_bson`: (required if valid is true) an (uppercase) big-endian hex representation of a BSON byte string.
+- `canonical_extjson`: (required if valid is true) string containing a Canonical Extended JSON document. Because this is
+  itself embedded as a *string* inside a JSON document, characters like quote and backslash are escaped.

source/bson-binary-vector/tests/float32.json

@@ -0,0 +1,45 @@
+{
+  "description": "Tests of Binary subtype 9, Vectors, with dtype FLOAT32",
+  "test_key": "vector",
+  "tests": [
+    {
+      "description": "Simple Vector FLOAT32",
+      "valid": true,
+      "vector": [127.0, 7.0],
+      "dtype_hex": "0x27",
+      "dtype_alias": "FLOAT32",
+      "padding": 0,
+      "canonical_bson": "1C00000005766563746F72000A0000000927000000FE420000E04000", 
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"JwAAAP5CAADgQA==\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Empty Vector FLOAT32",
+      "valid": true,
+      "vector": [],
+      "dtype_hex": "0x27",
+      "dtype_alias": "FLOAT32",
+      "padding": 0,
+      "canonical_bson": "1400000005766563746F72000200000009270000",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"JwA=\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Infinity Vector FLOAT32",
+      "valid": true,
+      "vector": ["-inf", 0.0, "inf"],
+      "dtype_hex": "0x27",
+      "dtype_alias": "FLOAT32",
+      "padding": 0,
+      "canonical_bson": "2000000005766563746F72000E000000092700000080FF000000000000807F00",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"JwAAAID/AAAAAAAAgH8=\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "FLOAT32 with padding",
+      "valid": false,
+      "vector": [127.0, 7.0],
+      "dtype_hex": "0x27",
+      "dtype_alias": "FLOAT32",
+      "padding": 3
+    }
+  ]
+}
+

source/bson-binary-vector/tests/int8.json

@@ -0,0 +1,59 @@
+{
+  "description": "Tests of Binary subtype 9, Vectors, with dtype INT8",
+  "test_key": "vector",
+  "tests": [
+    {
+      "description": "Simple Vector INT8",
+      "valid": true,
+      "vector": [127, 7],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 0,
+      "canonical_bson": "1600000005766563746F7200040000000903007F0700",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"AwB/Bw==\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Empty Vector INT8",
+      "valid": true,
+      "vector": [],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 0,
+      "canonical_bson": "1400000005766563746F72000200000009030000",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"AwA=\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Overflow Vector INT8",
+      "valid": false,
+      "vector": [128],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 0
+    },
+    {
+      "description": "Underflow Vector INT8",
+      "valid": false,
+      "vector": [-129],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 0
+    },
+    {
+      "description": "INT8 with padding",
+      "valid": false,
+      "vector": [127, 7],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 3
+    },
+    {
+      "description": "INT8 with float inputs",
+      "valid": false,
+      "vector": [127.77, 7.77],
+      "dtype_hex": "0x03",
+      "dtype_alias": "INT8",
+      "padding": 0
+    }
+  ]
+}
+

source/bson-binary-vector/tests/packed_bit.json

@@ -0,0 +1,53 @@
+{
+  "description": "Tests of Binary subtype 9, Vectors, with dtype PACKED_BIT",
+  "test_key": "vector",
+  "tests": [
+    {
+      "description": "Simple Vector PACKED_BIT",
+      "valid": true,
+      "vector": [127, 7],
+      "dtype_hex": "0x10",
+      "dtype_alias": "PACKED_BIT",
+      "padding": 0,
+      "canonical_bson": "1600000005766563746F7200040000000910007F0700",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"EAB/Bw==\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Empty Vector PACKED_BIT",
+      "valid": true,
+      "vector": [],
+      "dtype_hex": "0x10",
+      "dtype_alias": "PACKED_BIT",
+      "padding": 0,
+      "canonical_bson": "1400000005766563746F72000200000009100000",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"EAA=\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "PACKED_BIT with padding",
+      "valid": true,
+      "vector": [127, 7],
+      "dtype_hex": "0x10",
+      "dtype_alias": "PACKED_BIT",
+      "padding": 3,
+      "canonical_bson": "1600000005766563746F7200040000000910037F0700",
+      "canonical_extjson": "{\"vector\": {\"$binary\": {\"base64\": \"EAN/Bw==\", \"subType\": \"09\"}}}"
+    },
+    {
+      "description": "Overflow Vector PACKED_BIT",
+      "valid": false,
+      "vector": [256],
+      "dtype_hex": "0x10",
+      "dtype_alias": "PACKED_BIT",
+      "padding": 0
+    },
+    {
+      "description": "Underflow Vector PACKED_BIT",
+      "valid": false,
+      "vector": [-1],
+      "dtype_hex": "0x10",
+      "dtype_alias": "PACKED_BIT",
+      "padding": 0
+    }
+  ]
+}
+

source/bson-corpus/tests/binary.json

@@ -74,6 +74,36 @@
             "description": "$type query operator (conflicts with legacy $binary form with $type field)",
             "canonical_bson": "180000000378001000000010247479706500020000000000",
             "canonical_extjson": "{\"x\" : { \"$type\" : {\"$numberInt\": \"2\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector FLOAT32",
+            "canonical_bson": "170000000578000A0000000927000000FE420000E04000",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"JwAAAP5CAADgQA==\", \"subType\": \"09\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector INT8",
+            "canonical_bson": "11000000057800040000000903007F0700",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"AwB/Bw==\", \"subType\": \"09\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector PACKED_BIT",
+            "canonical_bson": "11000000057800040000000910007F0700",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"EAB/Bw==\", \"subType\": \"09\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector (Zero-length) FLOAT32",
+            "canonical_bson": "0F0000000578000200000009270000",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"JwA=\", \"subType\": \"09\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector (Zero-length) INT8",
+            "canonical_bson": "0F0000000578000200000009030000",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"AwA=\", \"subType\": \"09\"}}}"
+        },
+        {
+            "description": "subtype 0x09 Vector (Zero-length) PACKED_BIT",
+            "canonical_bson": "0F0000000578000200000009100000",
+            "canonical_extjson": "{\"x\": {\"$binary\": {\"base64\": \"EAA=\", \"subType\": \"09\"}}}"
         }
     ],
     "decodeErrors": [

source/index.md

@@ -1,6 +1,7 @@
 # MongoDB Specifications
 
 - [BSON Binary Subtype 6](client-side-encryption/subtype6.md)
+- [BSON Binary Subtype 9 - Vector](bson-binary-vector/bson-binary-vector.md)
 - [BSON Corpus](bson-corpus/bson-corpus.md)
 - [BSON Decimal128 Type Handling in Drivers](bson-decimal128/decimal128.md)
 - [Causal Consistency Specification](causal-consistency/causal-consistency.md)