Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRIVERS-716 Improved Bulk Write API #1534

Merged
merged 92 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from 75 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
fb27ce1
initial work
isabelatkinson Feb 2, 2024
0ede8ae
retryability
isabelatkinson Feb 14, 2024
afb1ab6
changelog
isabelatkinson Feb 14, 2024
e2cb197
transactions
isabelatkinson Feb 16, 2024
a24c7bd
unified test runner changes
isabelatkinson Feb 20, 2024
5a98994
add schema
isabelatkinson Feb 26, 2024
98fbfda
add batching prose tests
isabelatkinson Feb 26, 2024
7f59210
restructure spec
isabelatkinson Feb 27, 2024
dbb2083
various cleanup
isabelatkinson Feb 29, 2024
bc888f8
schema version updates
isabelatkinson Feb 29, 2024
b425c23
move run on requirements to top level
isabelatkinson Feb 29, 2024
d7294cd
always use verbose results
isabelatkinson Feb 29, 2024
82799a7
clean up yaml anchors
isabelatkinson Feb 29, 2024
905e2ee
clean up command started events
isabelatkinson Feb 29, 2024
981e6c3
move tests into crud directory
isabelatkinson Feb 29, 2024
03c3bde
fix typo
isabelatkinson Feb 29, 2024
01c9ae7
update some language
isabelatkinson Mar 1, 2024
fab1deb
self review changes
isabelatkinson Mar 1, 2024
7e1e897
language updates
isabelatkinson Mar 4, 2024
d907502
minor updates
isabelatkinson Mar 4, 2024
a4cf43f
review changes
isabelatkinson Mar 11, 2024
13f88a7
fix ordered
isabelatkinson Mar 11, 2024
ee27603
add future work
isabelatkinson Mar 11, 2024
94ed555
add errinfo
isabelatkinson Mar 12, 2024
b7a9d86
remove note escapes
isabelatkinson Mar 12, 2024
d557b6e
must -> should
isabelatkinson Mar 12, 2024
89c99f6
remove q
isabelatkinson Mar 14, 2024
82aa4f2
more review changes
isabelatkinson Mar 14, 2024
fb6007e
remove outdated test, add bypassDocumentValidation language
isabelatkinson Mar 15, 2024
c34e432
clarify
isabelatkinson Mar 15, 2024
0491cec
insert -> write
isabelatkinson Mar 15, 2024
c4ddf5b
clarify naming
isabelatkinson Mar 19, 2024
c308b1b
add network errors tests
isabelatkinson Apr 2, 2024
48b7cbf
add comment option
isabelatkinson Apr 2, 2024
d5d98f0
use exists for upsertedId
isabelatkinson Apr 2, 2024
43a4ffb
rename write concern error message
isabelatkinson Apr 2, 2024
d1e1a33
update upserted
isabelatkinson Apr 2, 2024
ec6863b
language
isabelatkinson Apr 2, 2024
7c8eb73
test lsid and txnNumber
isabelatkinson Apr 3, 2024
bd384b5
improve size batching length check
isabelatkinson Apr 4, 2024
1633501
add bypassdocumentvalidation false test
isabelatkinson Apr 4, 2024
7f31275
writeconcernerror batch prose test
isabelatkinson Apr 4, 2024
3ab6d00
writeerror batch prose test
isabelatkinson Apr 4, 2024
06cf1e3
command monitoring, cursor tests
isabelatkinson Apr 5, 2024
1677d57
killcursors
isabelatkinson Apr 5, 2024
c74634c
add pipeline tests
isabelatkinson Apr 5, 2024
7460b42
update vs. replace validation
isabelatkinson Apr 5, 2024
a83fe6b
add validation tests
isabelatkinson Apr 5, 2024
1077086
partial result note
isabelatkinson Apr 5, 2024
947004b
fix failed iteration test
isabelatkinson Apr 5, 2024
e6d8bc5
empty models test
isabelatkinson Apr 9, 2024
e2effb8
basic write concern test
isabelatkinson Apr 9, 2024
fa5395b
more write concern tests
isabelatkinson Apr 9, 2024
c231068
add retryWrites:false test
isabelatkinson Apr 9, 2024
bef5231
add unacknowledged test
isabelatkinson Apr 9, 2024
9b82f6f
reduce _id size
isabelatkinson Apr 9, 2024
6b4f623
iteration test updates
isabelatkinson Apr 11, 2024
1540e57
errorResponse
isabelatkinson Apr 11, 2024
aaf6a21
add files
isabelatkinson Apr 11, 2024
c5ba181
add unacknowledged monitoring test
isabelatkinson Apr 17, 2024
6995d24
add handshake error test
isabelatkinson Apr 17, 2024
e6f8100
operation ID server selection test
isabelatkinson Apr 17, 2024
c09c1d4
add pinning test
isabelatkinson Apr 17, 2024
fa4b434
add write concern transaction test
isabelatkinson Apr 17, 2024
919c506
another transaction test
isabelatkinson Apr 17, 2024
af0a16d
add csot test
isabelatkinson Apr 17, 2024
72eda1b
cleanup
isabelatkinson Apr 17, 2024
f389db6
add q&a
isabelatkinson Apr 17, 2024
8f54d52
nsInfo clarifications and test
isabelatkinson Apr 22, 2024
ec71c96
retry bulkWrite when getMore fails
isabelatkinson Apr 23, 2024
6cde925
Revert "retry bulkWrite when getMore fails"
isabelatkinson Apr 24, 2024
5993389
add future work section, rewrite size batch splitting rules
isabelatkinson Apr 24, 2024
ccb03ed
add ns batch split test
isabelatkinson Apr 26, 2024
834cca8
too large test
isabelatkinson Apr 26, 2024
80b7499
language
isabelatkinson Apr 26, 2024
2c9702d
language
isabelatkinson Apr 26, 2024
3fdd543
improve CSOT test
isabelatkinson Apr 30, 2024
7ef7c8c
jeremy language suggestions
isabelatkinson Apr 30, 2024
e133ed3
add oid to calculation info
isabelatkinson Apr 30, 2024
b9f39a6
update batching tests
isabelatkinson May 2, 2024
fb659af
remove auto-encryption support
isabelatkinson May 2, 2024
c12cf6c
server versions
isabelatkinson May 2, 2024
35b2250
Merge branch 'bulk-write' of github.com:isabelatkinson/specifications…
isabelatkinson May 2, 2024
a3d1652
update csot test formatting
isabelatkinson May 2, 2024
d23399c
1.20 -> 1.21
isabelatkinson May 8, 2024
91bec49
save files
isabelatkinson May 8, 2024
ebe592e
Merge branch 'master' into bulk-write
isabelatkinson May 8, 2024
cd64072
fix lint errors
isabelatkinson May 8, 2024
ae0d729
fix schema versions
isabelatkinson May 8, 2024
05852ad
json
isabelatkinson May 8, 2024
ad57a1e
bump schema version in makefile
isabelatkinson May 8, 2024
5a6532f
merge 1.20 and 1.21
isabelatkinson May 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion source/client-side-operations-timeout/tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ test MUST be unset using `internalClient` after the test has been executed. All
MUST be configured with read/write concern `majority`, read preference `primary`, and command monitoring enabled to
listen for `command_started` events.

### 1. Multi-batch writes
### 1. Multi-batch inserts

This test MUST only run against standalones on server versions 4.4 and higher. The `insertMany` call takes an
exceedingly long time on replicasets and sharded clusters. Drivers MAY adjust the timeouts used in this test to allow
Expand Down Expand Up @@ -598,6 +598,39 @@ Tests in this section MUST only run against replica sets and sharded clusters wi
1. `command_started` and `command_failed` events for an `insert` command.
2. `command_started` and `command_failed` events for an `abortTransaction` command.


### 11. Multi-batch bulkWrites

This test MUST only run against standalones on server versions 8.0 and higher. The `bulkWrite` call takes an
exceedingly long time on replicasets and sharded clusters. Drivers MAY adjust the timeouts used in this test to allow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would an insert batch of 50 1MB documents take an "exceedingly long time"? Also, if a fail point is being used on each command, couldn't you also get by using fewer, larger documents that split into 2+ commands and trigger the timeout?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, updated to insert a few large documents. I lifted most of this test from the existing insertMany test for consistency, but the actual writes happening shouldn't matter much as long as the same blockConnection is being used. Some basic local benchmarking of inserting a few large documents didn't show any time differences based on topology so I also removed that language.

Caveat: Kevin and I can't verify that this works as Rust and C both don't implement CSOT, so we'll need to wait for a driver that does have CSOT to implement this.

for differing bulk encoding performance.

1. Using `internalClient`, drop the `db.coll` collection.

2. Using `internalClient`, set the following fail point:

```javascript
{
configureFailPoint: "failCommand",
mode: {
times: 2
},
data: {
failCommands: ["bulkWrite"],
blockConnection: true,
blockTimeMS: 1010
}
}
```

3. Create a new MongoClient (referred to as `client`) with `timeoutMS=2000`.

4. Using `client`, insert 50 1-megabyte documents in a single `MongoClient.bulkWrite` call.

- Expect this to fail with a timeout error.

5. Verify that two `bulkWrite` commands were executed against `db.coll` as part of the `MongoClient.bulkWrite` call.

## Unit Tests

The tests enumerated in this section could not be expressed in either spec or prose format. Drivers SHOULD implement
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
description: "unacknowledged-client-bulkWrite"

schemaVersion: "1.0"

createEntities:
- client:
id: &client client
observeEvents:
- commandStartedEvent
- commandSucceededEvent
- commandFailedEvent
uriOptions:
w: 0
- database:
id: &database database
client: *client
databaseName: &databaseName command-monitoring-tests
- collection:
id: &collection collection
database: *database
collectionName: &collectionName test

initialData:
- collectionName: *collectionName
databaseName: *databaseName
documents:
- { _id: 1, x: 11 }
- { _id: 2, x: 22 }
- { _id: 3, x: 33 }

_yamlAnchors:
namespace: &namespace "command-monitoring-tests.test"

tests:
- description: 'A successful mixed client bulkWrite'
operations:
- object: *client
name: clientBulkWrite
arguments:
models:
- insertOne:
namespace: *namespace
document: { _id: 4, x: 44 }
- updateOne:
namespace: *namespace
filter: { _id: 3 }
update: { $set: { x: 333 } }
expectResult:
insertedCount:
$$unsetOrMatches: 0
upsertedCount:
$$unsetOrMatches: 0
matchedCount:
$$unsetOrMatches: 0
modifiedCount:
$$unsetOrMatches: 0
deletedCount:
$$unsetOrMatches: 0
insertResults:
$$unsetOrMatches: {}
updateResults:
$$unsetOrMatches: {}
deleteResults:
$$unsetOrMatches: {}
expectEvents:
-
client: *client
events:
- commandStartedEvent:
commandName: bulkWrite
databaseName: admin
command:
bulkWrite: 1
errorsOnly: true
ordered: true
ops:
- insert: 0
document: { _id: 4, x: 44 }
- update: 0
filter: { _id: 3 }
updateMods: { $set: { x: 333 } }
multi: false
nsInfo:
- ns: *namespace
- commandSucceededEvent:
commandName: bulkWrite
reply:
ok: 1
nInserted: { $$exists: false }
nMatched: { $$exists: false }
nModified: { $$exists: false }
nUpserted: { $$exists: false }
nDeleted: { $$exists: false }
Loading