CDRIVER-4363 add client bulk write #1590

kevinAlbs · 2024-05-01T17:25:57Z

Summary

This PR implements the MongoClient.bulkWrite API proposed in: mongodb/specifications#1534

RST documentation was omitted from this PR in case the API changes during review. I plan to add RST documentation in a future PR.

Commits are separated to ease filtering the JSON test files and minor drive-by improvements. See the bulk-write.md file in mongodb/specifications#1534 for the expected behavior.

Testing

JSON files are copied from the mongodb/specifications#1534. These were reviewed in the spec PR and I do not expect need another review. test-mongoc-crud.c includes the prose tests from the specification. test-mongoc-bulkwrite.c includes libmongoc-specific tests.

Background

Terms

The following terms are used in this PR description:

MongoCollection.bulkWrite refers to the existing API from the CRUD specification. The C driver implements in terms of mongoc_bulk_operation_t.
MongoClient.bulkWrite refers to the new API in the new Bulk Write specification. This PR implements in terms of the new mongoc_bulkwrite_t handle.

MongoCollection.bulkWrite splits write operations by type, and sends separate insert, update, and delete server commands. MongoCollection.bulkWrite writes to only one collection.

MongoClient.bulkWrite uses the bulkWrite server command. The bulkWrite command supports insert, update, and delete operations for multiple namespaces. The bulkWrite command is introduced in server 8.0. MongoClient.bulkWrite can write to multiple collections.

Both bulk write APIs require splitting large writes into separate commands to stay within server size limits. E.g. a call to MongoClient.bulkWrite with many writes may result in multiple bulkWrite commands sent.

Rationale

The following describes rationale behind some decisions in this PR:

Models

The specification defines model classes for each write. Here is the DeleteOneModel definition (with comments removed for brevity):

class DeleteOneModel implements WriteModel {
    filter: Document;
    collation: Optional<Document>;
    hint: Optional<Document | String>;
}

This PR represents the required fields in arguments, and groups optional fields in an opaque opts struct:

MONGOC_EXPORT (bool)
mongoc_bulkwrite_append_deleteone (mongoc_bulkwrite_t *self,
                                   const char *ns,
                                   const bson_t *filter,
                                   mongoc_bulkwrite_deleteoneopts_t *opts /* May be NULL */,
                                   bson_error_t *error);

I thought this was a reasonable compromise between matching the spec and existing C driver patterns. If the spec later extends DeleteOneModel, mongoc_bulkwrite_deleteoneopts_t can be extended.

The existing C driver implementation of MongoCollection.bulkWrite uses bson_t for options:

MONGOC_EXPORT (bool)
mongoc_bulk_operation_remove_one_with_opts (mongoc_bulk_operation_t *bulk,
                                            const bson_t *selector,
                                            const bson_t *opts,
                                            bson_error_t *error); /* OUT */

However, I think a typed mongoc_bulkwrite_deleteoneopts_t is preferable to bson_t. It may ease development with compile time type checks and auto complete.

Naming

This PR uses a slightly different API naming pattern. Parts are logically separated by an underscore. Example: mongoc_bulkwriteopts_set_bypassdocumentvalidation.
This differs slightly from the established pattern of separating each word. Example in current API: mongoc_bulk_operation_set_bypass_document_validation.
I find the logical separation easier to read.

Validation

The existing C driver implementation of MongoCollection.bulkWrite includes default client-side validation:

validate: Construct a bitwise-or of all desired bson_validate_flags_t. Set to false to skip client-side validation of the provided BSON documents.

The default flags for writes are all the same:

const bson_validate_flags_t _mongoc_default_insert_vflags =
   BSON_VALIDATE_UTF8 | BSON_VALIDATE_UTF8_ALLOW_NULL |
   BSON_VALIDATE_EMPTY_KEYS;

UTF-8 validation appears broken. Passing invalid UTF-8 for string values does not result in error. See: CDRIVER-4448. The default validation only enables BSON_VALIDATE_EMPTY_KEYS to check there are no empty keys (e.g. {"": "foo"}).

CDRIVER-3731 is a request to configure validation at a broader scope. Quoting jeroen/mongolite#206:

It probably makes sense to disable this globally

The C driver benchmarks disable the default validation. I re-enabled the validation and reran the benchmark. This resulted in significant performance drops in the base comparison:

Since UTF-8 validation is already broken, there are requests to disable, validation negatively impacts performance, and it is not a spec requirement, this PR excludes default validation in MongoClient.bulkWrite.

This is a drive-by fix to fix the referenced payload numbers.

Otherwise, if the type mismatches, the test runner aborts with an error: "expected int64, int32, or double [...]" that does not identify the field.

To do exact BSON matches

To ease reading error message

include some tests of basic usage and libmongoc-specific behavior. Specification and prose tests test driver-agnostic behavior.

To support `w=majority`

kkloberdanz · 2024-05-03T16:27:19Z

src/libmongoc/tests/unified/result.c

+      if (write_errors) {
+         if (write_errors && !result->write_errors) {


I believe the write_errors in the nested if is redundant.

kkloberdanz · 2024-05-06T15:18:01Z

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

+      bson_oid_init (&oid, NULL);
+      BSON_ASSERT (BSON_APPEND_OID (&tmp, "_id", &oid));
+      BSON_ASSERT (bson_concat (&tmp, document));
+      BSON_ASSERT (bson_append_document (&op, "document", 8, &tmp));


Rather than hardcoding a string length, I find it more maintainable to use something like strlen("document") instead. The compiler should optimize away the call to strlen. See this example on godbolt https://godbolt.org/z/bhzoddYKo.

PHP internals often does sizeof(<str literal>) - 1, but strlen() is certainly more concise if it gets compiled the same.

There also used to be a PHP macro for expanding a string literal into two arguments (the string and length), but it had a cryptic name and casual readers might not realize it expands and assume a parameter is missing. If that sounds useful, readability could probably be addressed with a more verbose name (e.g. STR_AND_LEN()). The main benefit is avoiding duplication (be it the original string or its length).

The compiler should optimize away the call to strlen.

Neat, thank you for the example. Updated to use the BSON_APPEND_* macros to remove the hard coded lengths. The macros add a strlen call.

kkloberdanz · 2024-05-07T15:27:18Z

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

+}
+
+mongoc_bulkwritereturn_t
+mongoc_bulkwrite_execute (mongoc_bulkwrite_t *self, const mongoc_bulkwriteopts_t *opts)


This function is nearly 600 lines long. Is it possible to break this function up into smaller functions? The general rule of thumb that I use is to make functions that fit in a single screen size without needing to scroll (approximately 80 lines).

Updated to split into two new helpers: _bulkwritereturn_apply_result and _bulkwritereturn_apply_reply.

An attempt to more aggressively split did not improve readability IMO. I think separate functions are helpful when it is "easy" to reason about what state is changed. Result handling seemed like an improvement, since only the mongoc_bulkwritereturn_t is updated.

Though this function is still long, block scopes with comments were used to logically group. Code folding can collapse blocks when reading:

kkloberdanz · 2024-05-08T20:28:27Z

src/libmongoc/tests/test-mongoc-crud.c

+   tf->maxMessageSizeBytes = sl.maxMessageSizeBytes;
+   tf->maxBsonObjectSize = sl.maxBsonObjectSize;
+
+   const int32_t opsBytes = tf->maxMessageSizeBytes - 1122;


Please comment the meaning of 1122 or create a constant with a meaningful name and assign this value to it. Please do the same for other locations where magic numbers are used, e.g., 57, 217, etc.

The values are copied from the prose test. The prose test describes the calculation.

Added a comment referring to the prose tests.

src/libmongoc/tests/unified/runner.c

src/libmongoc/tests/TestSuite.c

src/libmongoc/tests/TestSuite.h

src/libmongoc/src/mongoc/mongoc-write-command.c

src/libmongoc/tests/bsonutil/bson-val.c

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

Replace with existing `mongoc_optional_t`.

Find once and store the offset.

Array filters do not apply to replace. Remove to match spec.

The return was not being set. Callers can determine if the write was acknowledged by checking if `mongoc_bulkwritereturn_t::res` is NULL.

Only select a new stream if another batch is needed.

Reduces repeated assignment

Without the error propagation change, the comparison of int to string aborts (rather than returning an error).

Fix updateResults and deleteResults to return values as int64 to match spec.

Use `lookup_as_int64` for `ok` field. Server's `BulkWriteReplyItem` appends as double. Support other numerics for future proofing.

…ply`

…sult`

kevinAlbs

Latest changes verified with this patch build.

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

kevinAlbs · 2024-05-10T17:36:40Z

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

+      bson_oid_init (&oid, NULL);
+      BSON_ASSERT (BSON_APPEND_OID (&tmp, "_id", &oid));
+      BSON_ASSERT (bson_concat (&tmp, document));
+      BSON_ASSERT (bson_append_document (&op, "document", 8, &tmp));


The compiler should optimize away the call to strlen.

Neat, thank you for the example. Updated to use the BSON_APPEND_* macros to remove the hard coded lengths. The macros add a strlen call.

src/libmongoc/src/mongoc/mongoc-write-command.c

src/libmongoc/tests/TestSuite.c

src/libmongoc/tests/bsonutil/bson-val.c

kevinAlbs · 2024-05-14T14:04:33Z

src/libmongoc/tests/test-mongoc-crud.c

+   tf->maxMessageSizeBytes = sl.maxMessageSizeBytes;
+   tf->maxBsonObjectSize = sl.maxBsonObjectSize;
+
+   const int32_t opsBytes = tf->maxMessageSizeBytes - 1122;


The values are copied from the prose test. The prose test describes the calculation.

Added a comment referring to the prose tests.

src/libmongoc/tests/unified/runner.c

jmikola

Reviewed requested changes from my previous review. Note that I did not diligently examine the test changes (just the public API we'd end up using in PHPC).

kkloberdanz

LGTM!

adriandole · 2024-05-16T02:31:19Z

src/libmongoc/src/mongoc/mongoc-bulkwrite.c

+mongoc_bulkwriteopts_set_let (mongoc_bulkwriteopts_t *self, const bson_t *let)
+{
+   BSON_ASSERT_PARAM (self);
+   BSON_ASSERT (let || true);


I'd rather not introduce more instances of this funny line. I think we need something like:

#define BSON_OPTIONAL_PARAM (x) (void)(x)

Added BSON_OPTIONAL_PARAM.

kevinAlbs added 5 commits April 30, 2024 08:57

fix comment in _mongoc_write_opmsg

6535664

This is a drive-by fix to fix the referenced payload numbers.

quote key in match error

460bd8d

propagate error for numeric mismatch

cc19e7a

Otherwise, if the type mismatches, the test runner aborts with an error: "expected int64, int32, or double [...]" that does not identify the field.

add ASSERT_EQUAL_BSON

e1298a0

To do exact BSON matches

align print on match failure

8bbaecc

To ease reading error message

kevinAlbs mentioned this pull request May 1, 2024

add stub API for client bulk write #1574

Closed

kevinAlbs added 6 commits May 1, 2024 15:28

implement client bulk write

2e479db

include some tests of basic usage and libmongoc-specific behavior. Specification and prose tests test driver-agnostic behavior.

add example use

c8e4763

add prose tests

c1dccd1

add specification tests

43d3e65

support string w URI option in test runner

6f44f72

To support `w=majority`

update unified test runner

b031041

kevinAlbs force-pushed the try2.rebased.impl.C4363 branch from 137db61 to b031041 Compare May 1, 2024 19:29

kevinAlbs marked this pull request as ready for review May 1, 2024 20:03

kevinAlbs requested review from kkloberdanz, adriandole and vector-of-bool May 1, 2024 20:03

kkloberdanz suggested changes May 8, 2024

View reviewed changes

jmikola reviewed May 9, 2024

View reviewed changes

kevinAlbs added 6 commits May 10, 2024 09:17

remove redundant write_errors check

8a4cda6

remove mongoc_opt_bool_t

59c4446

Replace with existing `mongoc_optional_t`.

add ERROR_IF_EXECUTED macro

fd788ef

use BSON_APPEND_* macros instead of hard coded lengths

4717551

remove repeated call to bson_iter_init_find

86225a2

Find once and store the offset.

avoid repeated calls to _mongoc_document_is_pipeline

26ac0ff

kevinAlbs removed the request for review from vector-of-bool May 10, 2024 20:32

kevinAlbs added 4 commits May 11, 2024 14:26

remove mongoc_bulkwrite_replaceoneopts_set_arrayfilters

1c18ae5

Array filters do not apply to replace. Remove to match spec.

remove mongoc_bulkwriteresult_acknowledged

c150e70

The return was not being set. Callers can determine if the write was acknowledged by checking if `mongoc_bulkwritereturn_t::res` is NULL.

expect non-NULL error

6886add

remove server ID assignment

d5f0c6d

kevinAlbs added 15 commits May 13, 2024 11:25

assign server ID on retry

fbf0678

test upsert with null _id

00ceb67

check for stream validity before sending command

eff456e

Only select a new stream if another batch is needed.

add missing assignment of server ID

960723e

assign server ID shortly before end

77b30ed

Reduces repeated assignment

add missing _bulkwriteexception_set_error_reply

e5c978d

clarify calculation of opmsg_overhead for collection bulk write

24904ae

test matching with numerics

93feb70

Without the error propagation change, the comparison of int to string aborts (rather than returning an error).

add lookup_as_int64, and fix verbose results

a75a4eb

Fix updateResults and deleteResults to return values as int64 to match spec.

remove lookup_double

60ca972

Use `lookup_as_int64` for `ok` field. Server's `BulkWriteReplyItem` appends as double. Support other numerics for future proofing.

refer to spec for prose test 11 values

0394f43

simplify check_schema_version

b4182ec

split handling top-level reply fields into `_bulkwritereturn_apply_re…

35f2bfc

…ply`

split handling top-level reply fields into `_bulkwritereturn_apply_re…

3abda24

…sult`

relocate comment

65a6869

kevinAlbs commented May 14, 2024

View reviewed changes

kevinAlbs requested review from kkloberdanz and jmikola May 14, 2024 19:43

jmikola approved these changes May 15, 2024

View reviewed changes

kkloberdanz approved these changes May 15, 2024

View reviewed changes

adriandole approved these changes May 16, 2024

View reviewed changes

kevinAlbs added 2 commits May 16, 2024 09:29

add BSON_OPTIONAL_PARAM macro

b5cb200

replace BSON_ASSERT (X || true) with BSON_OPTIONAL_PARAM (X)

f1980a5

kevinAlbs merged commit 84728bd into mongodb:master May 16, 2024
41 of 45 checks passed

This was referenced May 16, 2024

CDRIVER-4363 document mongoc_bulkwrite_t #1602

Merged

CDRIVER-5560 add server 8.0 tasks #1603

Merged

eramongodb mentioned this pull request Jul 10, 2024

Address -Wunused-parameter warnings #1671

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CDRIVER-4363 add client bulk write #1590

CDRIVER-4363 add client bulk write #1590

kevinAlbs commented May 1, 2024 •

edited

Loading

kkloberdanz May 3, 2024

kkloberdanz May 6, 2024

jmikola May 9, 2024

kevinAlbs May 10, 2024

kkloberdanz May 7, 2024

kevinAlbs May 13, 2024

kkloberdanz May 8, 2024

kevinAlbs May 14, 2024

kevinAlbs left a comment

kevinAlbs May 10, 2024

kevinAlbs May 14, 2024

jmikola left a comment

kkloberdanz left a comment

adriandole May 16, 2024

kevinAlbs May 16, 2024

		if (write_errors) {
		if (write_errors && !result->write_errors) {

CDRIVER-4363 add client bulk write #1590

CDRIVER-4363 add client bulk write #1590

Conversation

kevinAlbs commented May 1, 2024 • edited Loading

Summary

Testing

Background

Terms

Rationale

Models

Naming

Validation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinAlbs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmikola left a comment

Choose a reason for hiding this comment

kkloberdanz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinAlbs commented May 1, 2024 •

edited

Loading