Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRIVERS-716 Improved Bulk Write API #1534

Merged
merged 92 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
fb27ce1
initial work
isabelatkinson Feb 2, 2024
0ede8ae
retryability
isabelatkinson Feb 14, 2024
afb1ab6
changelog
isabelatkinson Feb 14, 2024
e2cb197
transactions
isabelatkinson Feb 16, 2024
a24c7bd
unified test runner changes
isabelatkinson Feb 20, 2024
5a98994
add schema
isabelatkinson Feb 26, 2024
98fbfda
add batching prose tests
isabelatkinson Feb 26, 2024
7f59210
restructure spec
isabelatkinson Feb 27, 2024
dbb2083
various cleanup
isabelatkinson Feb 29, 2024
bc888f8
schema version updates
isabelatkinson Feb 29, 2024
b425c23
move run on requirements to top level
isabelatkinson Feb 29, 2024
d7294cd
always use verbose results
isabelatkinson Feb 29, 2024
82799a7
clean up yaml anchors
isabelatkinson Feb 29, 2024
905e2ee
clean up command started events
isabelatkinson Feb 29, 2024
981e6c3
move tests into crud directory
isabelatkinson Feb 29, 2024
03c3bde
fix typo
isabelatkinson Feb 29, 2024
01c9ae7
update some language
isabelatkinson Mar 1, 2024
fab1deb
self review changes
isabelatkinson Mar 1, 2024
7e1e897
language updates
isabelatkinson Mar 4, 2024
d907502
minor updates
isabelatkinson Mar 4, 2024
a4cf43f
review changes
isabelatkinson Mar 11, 2024
13f88a7
fix ordered
isabelatkinson Mar 11, 2024
ee27603
add future work
isabelatkinson Mar 11, 2024
94ed555
add errinfo
isabelatkinson Mar 12, 2024
b7a9d86
remove note escapes
isabelatkinson Mar 12, 2024
d557b6e
must -> should
isabelatkinson Mar 12, 2024
89c99f6
remove q
isabelatkinson Mar 14, 2024
82aa4f2
more review changes
isabelatkinson Mar 14, 2024
fb6007e
remove outdated test, add bypassDocumentValidation language
isabelatkinson Mar 15, 2024
c34e432
clarify
isabelatkinson Mar 15, 2024
0491cec
insert -> write
isabelatkinson Mar 15, 2024
c4ddf5b
clarify naming
isabelatkinson Mar 19, 2024
c308b1b
add network errors tests
isabelatkinson Apr 2, 2024
48b7cbf
add comment option
isabelatkinson Apr 2, 2024
d5d98f0
use exists for upsertedId
isabelatkinson Apr 2, 2024
43a4ffb
rename write concern error message
isabelatkinson Apr 2, 2024
d1e1a33
update upserted
isabelatkinson Apr 2, 2024
ec6863b
language
isabelatkinson Apr 2, 2024
7c8eb73
test lsid and txnNumber
isabelatkinson Apr 3, 2024
bd384b5
improve size batching length check
isabelatkinson Apr 4, 2024
1633501
add bypassdocumentvalidation false test
isabelatkinson Apr 4, 2024
7f31275
writeconcernerror batch prose test
isabelatkinson Apr 4, 2024
3ab6d00
writeerror batch prose test
isabelatkinson Apr 4, 2024
06cf1e3
command monitoring, cursor tests
isabelatkinson Apr 5, 2024
1677d57
killcursors
isabelatkinson Apr 5, 2024
c74634c
add pipeline tests
isabelatkinson Apr 5, 2024
7460b42
update vs. replace validation
isabelatkinson Apr 5, 2024
a83fe6b
add validation tests
isabelatkinson Apr 5, 2024
1077086
partial result note
isabelatkinson Apr 5, 2024
947004b
fix failed iteration test
isabelatkinson Apr 5, 2024
e6d8bc5
empty models test
isabelatkinson Apr 9, 2024
e2effb8
basic write concern test
isabelatkinson Apr 9, 2024
fa5395b
more write concern tests
isabelatkinson Apr 9, 2024
c231068
add retryWrites:false test
isabelatkinson Apr 9, 2024
bef5231
add unacknowledged test
isabelatkinson Apr 9, 2024
9b82f6f
reduce _id size
isabelatkinson Apr 9, 2024
6b4f623
iteration test updates
isabelatkinson Apr 11, 2024
1540e57
errorResponse
isabelatkinson Apr 11, 2024
aaf6a21
add files
isabelatkinson Apr 11, 2024
c5ba181
add unacknowledged monitoring test
isabelatkinson Apr 17, 2024
6995d24
add handshake error test
isabelatkinson Apr 17, 2024
e6f8100
operation ID server selection test
isabelatkinson Apr 17, 2024
c09c1d4
add pinning test
isabelatkinson Apr 17, 2024
fa4b434
add write concern transaction test
isabelatkinson Apr 17, 2024
919c506
another transaction test
isabelatkinson Apr 17, 2024
af0a16d
add csot test
isabelatkinson Apr 17, 2024
72eda1b
cleanup
isabelatkinson Apr 17, 2024
f389db6
add q&a
isabelatkinson Apr 17, 2024
8f54d52
nsInfo clarifications and test
isabelatkinson Apr 22, 2024
ec71c96
retry bulkWrite when getMore fails
isabelatkinson Apr 23, 2024
6cde925
Revert "retry bulkWrite when getMore fails"
isabelatkinson Apr 24, 2024
5993389
add future work section, rewrite size batch splitting rules
isabelatkinson Apr 24, 2024
ccb03ed
add ns batch split test
isabelatkinson Apr 26, 2024
834cca8
too large test
isabelatkinson Apr 26, 2024
80b7499
language
isabelatkinson Apr 26, 2024
2c9702d
language
isabelatkinson Apr 26, 2024
3fdd543
improve CSOT test
isabelatkinson Apr 30, 2024
7ef7c8c
jeremy language suggestions
isabelatkinson Apr 30, 2024
e133ed3
add oid to calculation info
isabelatkinson Apr 30, 2024
b9f39a6
update batching tests
isabelatkinson May 2, 2024
fb659af
remove auto-encryption support
isabelatkinson May 2, 2024
c12cf6c
server versions
isabelatkinson May 2, 2024
35b2250
Merge branch 'bulk-write' of github.com:isabelatkinson/specifications…
isabelatkinson May 2, 2024
a3d1652
update csot test formatting
isabelatkinson May 2, 2024
d23399c
1.20 -> 1.21
isabelatkinson May 8, 2024
91bec49
save files
isabelatkinson May 8, 2024
ebe592e
Merge branch 'master' into bulk-write
isabelatkinson May 8, 2024
cd64072
fix lint errors
isabelatkinson May 8, 2024
ae0d729
fix schema versions
isabelatkinson May 8, 2024
05852ad
json
isabelatkinson May 8, 2024
ad57a1e
bump schema version in makefile
isabelatkinson May 8, 2024
5a6532f
merge 1.20 and 1.21
isabelatkinson May 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
721 changes: 721 additions & 0 deletions source/crud/bulk-write.md

Large diffs are not rendered by default.

117 changes: 117 additions & 0 deletions source/crud/tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,3 +201,120 @@ CommandSucceededEvents. Then, insert an invalid document (e.g. `{x: 1}`) and ass
code is `121` (i.e. DocumentValidationFailure), and that its `details` property is accessible. Additionally, assert that
a CommandSucceededEvent was observed and that the `writeErrors[0].errInfo` field in the response document matches the
WriteError's `details` property.

### 3. `MongoClient.bulkWrite` handles a `writeModels` input with greater than `maxWriteBatchSize` operations

Test that `MongoClient.bulkWrite` properly handles `writeModels` inputs containing a number of writes greater than
`maxWriteBatchSize`.

This test must only be run on 8.0+ servers.

Construct a `MongoClient` (referred to as `client`) with
[command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) enabled to observe
CommandStartedEvents. Perform a `hello` command using `client` and record the `maxWriteBatchSize` value contained in the
response. Then, construct the following write model (referred to as `writeModel`):

```json
InsertOne: {
"namespace": "db.coll",
"document": { "a": "b" }
}
```

kevinAlbs marked this conversation as resolved.
Show resolved Hide resolved
Construct a list of write models (referred to as `models`) with `writeModel` repeated `maxWriteBatchSize + 1` times.
Execute `bulkWrite` on `client` with `models`. Assert that the bulk write succeeds and returns a `BulkWriteResult`
with an `insertedCount` value of `maxWriteBatchSize + 1`.

Assert that two CommandStartedEvents (referred to as `firstEvent` and `secondEvent`) were observed for the `bulkWrite` command.
Assert that the length of `firstEvent.command.ops` is `maxWriteBatchSize`. Assert that the length of `secondEvent.command.ops`
is 1. If the driver exposes `operationId`s in its CommandStartedEvents, assert that `firstEvent.operationId` is equal to
`secondEvent.operationId`.

### 4. `MongoClient.bulkWrite` handles a `writeModels` input larger than `maxBsonObjectSize`

Test that `MongoClient.bulkWrite` properly handles a `writeModels` input which constructs an `ops` array larger than
`maxBsonObjectSize`.

This test must only be run on 8.0+ servers.

Construct a `MongoClient` (referred to as `client`) with
[command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) enabled to observe
CommandStartedEvents. Perform a `hello` command using `client` and record the `maxBsonObjectSize` value contained in the
response. Then, construct the following document (referred to as `document`):

```json
{
"a": "b".repeat(maxBsonObjectSize / 2)
}
```

Construct the following list of write models (referred to as `models`):

```json
[
InsertOne: {
"namespace": "db.coll",
"document": document
},
InsertOne: {
"namespace": "db.coll",
"document": document
}
]
```

Execute `bulkWrite` on `client` with `models`. Assert that the bulk write succeeds and returns a `BulkWriteResult`
with an `insertedCount` value of 2. Then make the following assertions based on whether or not the driver uses document
sequences (`OP_MSG` payload type 1) with `MongoClient.bulkWrite`:

#### With document sequences

Assert that a single CommandStartedEvent (referred to as `event`) was observed for the `bulkWrite` command. Assert that
the length of `event.command.ops` is 2.

#### Without document sequences

Assert that two CommandStartedEvents (referred to as `firstEvent` and `secondEvent`) were observed for the `bulkWrite`
command. For both `firstEvent` and `secondEvent`, assert that the length of `command.ops` is 1. If the driver exposes
`operationId`s in its CommandStartedEvents, assert that `firstEvent.operationId` is equal to `secondEvent.operationId`.

### 5. `MongoClient.bulkWrite` with document sequences handles a `bulkWrite` message larger than `maxMessageSizeBytes`

Test that `MongoClient.bulkWrite` properly handles a `writeModels` input for which the sum of the models' entries in the
`ops` array exceeds `maxMessageSizeBytes`. Drivers that do not use document sequences (`OP_MSG` payload type 1) for bulk
writes should not implement this test.

This test must only be run on 8.0+ servers.

Construct a `MongoClient` (referred to as `client`) with
[command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) enabled to observe
CommandStartedEvents. Perform a `hello` command using `client` and record the following values from the response:
`maxBsonObjectSize` and `maxMessageSizeBytes`. Then, construct the following document (referred to as `document`):

```json
{
"a": "b".repeat(maxBsonObjectSize - 500)
}
```

Construct the following write model (referred to as `model`):

```json
InsertOne: {
"namespace": "db.coll",
"document": document
}
```

kevinAlbs marked this conversation as resolved.
Show resolved Hide resolved
Use the following calculation to determine the number of inserts that should be provided to `MongoClient.bulkWrite`:
`maxMessageSizeBytes / maxBsonObjectSize + 1` (referred to as `numModels`). This number ensures that the inserts
provided to `MongoClient.bulkWrite` will require multiple `bulkWrite` commands to be sent to the server.

Construct as list of write models (referred to as `models`) with `model` repeated `numModels` times. Then execute
`bulkWrite` on `client` with `models`. Assert that the bulk write succeeds and returns a `BulkWriteResult` with
an `insertedCount` value of `numModels`.

Assert that two CommandStartedEvents (referred to as `firstEvent` and `secondEvent`) were observed. Record the length of
`firstEvent.command.ops` as `firstOpsLen`. Record the length of `secondEvent.command.ops` as `secondOpsLen`. Assert that
`firstOpsLen + secondOpsLen` is equal to `numModels`. If the driver exposes `operationId`s in its CommandStartedEvents,
assert that `firstEvent.operationId` is equal to `secondEvent.operationId`.
267 changes: 267 additions & 0 deletions source/crud/tests/unified/client-bulkWrite-delete-options.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading