From 780d296b904f24fe086adad71d46ef8bdec18f69 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 10:07:47 -0500 Subject: [PATCH 01/18] DRIVERS-2870 Fix content of retryable-writes-test readme --- source/retryable-writes/tests/README.md | 204 ++++++++++++++++++++++-- 1 file changed, 192 insertions(+), 12 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 749d29ee54..cd8320ce03 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -71,7 +71,7 @@ Drivers should also assert that command documents are properly constructed with on whether the write operation is supported. [Command Logging and Monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) may be used to check for the presence of a `txnNumber` field in the command document. Note that command documents may always include an -`lsid` field per the [Driver Session](../../sessions/driver-sessions.md) specification. +`lsid` field per the [Driver Session](../../sessions/driver-sessions.rst) specification. These tests may be run against both a replica set and shard cluster. @@ -106,17 +106,197 @@ Drivers should test that transactions IDs are always included in commands for su The following tests ensure that retryable writes work properly with replica sets and sharded clusters. -1. Test that retryable writes raise an exception when using the MMAPv1 storage engine. For this test, execute a write - operation, such as `insertOne`, which should generate an exception. Assert that the error message is the replacement - error message: +- Test that retryable writes raise an exception when using the MMAPv1 storage engine. For this test, execute a write + operation, such as `insertOne`, which should generate an exception. Assert that the error message is the replacement + error message: - ``` - This MongoDB deployment does not support retryable writes. Please add - retryWrites=false to your connection string. - ``` + ``` + This MongoDB deployment does not support retryable writes. Please add + retryWrites=false to your connection string. + ``` - and the error code is 20. + and the error code is 20. - [!NOTE] - storage engine in use MAY skip this test for sharded clusters, since `mongos` does not report this information in its - `serverStatus` response. + > [!NOTE] + > Drivers that rely on `serverStatus` to determine the storage engine in use MAY skip this test for sharded clusters, + > since `mongos` does not report this information in its `serverStatus` response. + +- Test that drivers properly retry after encountering PoolClearedErrors. This test MUST be implemented by any driver + that implements the CMAP specification. + + This test requires MongoDB 4.3.4+ for both the `errorLabels` and `blockConnection` fail point options. + + - Create a client with maxPoolSize=1 and retryWrites=true. If testing against a sharded deployment, be sure to connect + to only a single mongos. + + - Enable the following failpoint: + + ```javascript + { + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 91, + blockConnection: true, + blockTimeMS: 1000, + errorLabels: ["RetryableWriteError"] + } + } + ``` + + - Start two threads and attempt to perform an `insertOne` simultaneously on both. + + - Verify that both `insertOne` attempts succeed. + + - Via CMAP monitoring, assert that the first check out succeeds. + + - Via CMAP monitoring, assert that a PoolClearedEvent is then emitted. + + - Via CMAP monitoring, assert that the second check out then fails due to a connection error. + + - Via Command Monitoring, assert that exactly three `insert` CommandStartedEvents were observed in total. + + - Disable the failpoint. + +- Test that drivers return the original error after encountering a WriteConcernError with a RetryableWriteError label. + This test MUST + + - be implemented by any driver that implements the Command Monitoring specification, + + - only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. + + - be run against server versions 6.0 and above. + + Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the + subsequent retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or + write a unit test to cover the same sequence of events. + + - Create a client with `retryWrites=true`. + + - Configure a fail point with error code `91` (ShutdownInProgress): + + ```javascript + { + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorLabels: ["RetryableWriteError"], + writeConcernError: { code: 91 } + } + } + ``` + + - Via the command monitoring CommandSucceededEvent, configure a fail point with error code `10107` + (NotWritablePrimary) and a NoWritesPerformed label: + + ```javascript + { + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorCode: 10107, + errorLabels: ["RetryableWriteError", "NoWritesPerformed"] + } + } + ``` + + Drivers SHOULD only configure the `10107` fail point command if the the succeeded event is for the `91` error + configured in step 2. + + - Attempt an `insertOne` operation on any record for any database and collection. For the resulting error, assert that + the associated error code is `91`. + + - Disable the fail point: + + ```javascript + { + configureFailPoint: "failCommand", + mode: "off" + } + ``` + +- Test that in a sharded cluster writes are retried on a different mongos when one is available. This test MUST be + executed against a sharded cluster that has at least two mongos instances, supports `retryWrites=true`, has enabled + the `configureFailPoint` command, and supports the `errorLabels` field (MongoDB 4.3.1+). + + Note: this test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior + intended to be tested) from "retry on a different mongos due to normal SDAM randomized suitable server selection". + Verify relevant code paths are correctly executed by the tests using external means such as a logging, debugger, code + coverage tool, etc. + + - Create two clients `s0` and `s1` that each connect to a single mongos from the sharded cluster. They must not + connect to the same mongos. + + - Configure the following fail point for both `s0` and `s1`: + + ```javascript + { + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 6, + errorLabels: ["RetryableWriteError"] + } + } + ``` + + - Create a client `client` with `retryWrites=true` that connects to the cluster using the same two mongoses as `s0` + and `s1`. + + - Enable failed command event monitoring for `client`. + + - Execute an `insert` command with `client`. Assert that the command failed. + + - Assert that two failed command events occurred. Assert that the failed command events occurred on different + mongoses. + + - Disable the fail points on both `s0` and `s1`. + +## Changelog + +- 2024-05-30: Migrated from reStructuredText to Markdown. + +- 2024-02-27: Convert legacy retryable writes tests to unified format. + +- 2024-02-21: Update prose test 4 and 5 to workaround SDAM behavior preventing\ + execution of deprioritization code + paths. + +- 2024-01-05: Fix typo in prose test title. + +- 2024-01-03: Note server version requirements for fail point options and revise\ + tests to specify the `errorLabels` + option at the top-level instead of within `writeConcernError`. + +- 2023-08-26: Add prose tests for retrying in a sharded cluster. + +- 2022-08-30: Add prose test verifying correct error handling for errors with\ + the NoWritesPerformed label, which is to + return the original error. + +- 2022-04-22: Clarifications to `serverless` and `useMultipleMongoses`. + +- 2021-08-27: Add `serverless` to `runOn`. Clarify behavior of\ + `useMultipleMongoses` for `LoadBalanced` topologies. + +- 2021-04-23: Add `load-balanced` to test topology requirements. + +- 2021-03-24: Add prose test verifying `PoolClearedErrors` are retried. + +- 2019-10-21: Add `errorLabelsContain` and `errorLabelsContain` fields to\ + `result` + +- 2019-08-07: Add Prose Tests section + +- 2019-06-07: Mention $merge stage for aggregate alongside $out + +- 2019-03-01: Add top-level `runOn` field to denote server version and/or\ + topology requirements requirements for the + test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now expressed within + `runOn` elements. + + Add test-level `useMultipleMongoses` field. From 73f19234b65ccb948118ca5cd240d128506cf2f0 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:27:57 -0500 Subject: [PATCH 02/18] address review --- .pre-commit-config.yaml | 16 +- source/retryable-writes/tests/README.md | 241 ++++++++++++------------ 2 files changed, 131 insertions(+), 126 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index db6ed1322d..92862432fd 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -12,6 +12,14 @@ repos: #- id: trailing-whitespace #- id: check-json +- repo: https://github.com/executablebooks/mdformat + rev: 0.7.17 + hooks: + - id: mdformat + args: ["--wrap=120", "--number"] + additional_dependencies: + [mdformat-gfm, mdformat-frontmatter, mdformat-footnote, mdformat-gfm-alerts] + # We use the Python version instead of the original version which seems to require Docker # https://github.com/koalaman/shellcheck-precommit - repo: https://github.com/shellcheck-py/shellcheck-py @@ -33,14 +41,6 @@ repos: types: [markdown] entry: bash scripts/check_anchors/check-anchors.sh language: node - -- repo: https://github.com/executablebooks/mdformat - rev: 0.7.17 - hooks: - - id: mdformat - args: ["--wrap=120", "--number"] - additional_dependencies: - [mdformat-gfm, mdformat-frontmatter, mdformat-footnote, mdformat-gfm-alerts] - repo: https://github.com/tcort/markdown-link-check rev: v3.11.2 diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index cd8320ce03..56927de35e 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -71,7 +71,7 @@ Drivers should also assert that command documents are properly constructed with on whether the write operation is supported. [Command Logging and Monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) may be used to check for the presence of a `txnNumber` field in the command document. Note that command documents may always include an -`lsid` field per the [Driver Session](../../sessions/driver-sessions.rst) specification. +`lsid` field per the [Driver Session](../../sessions/driver-sessions.md) specification. These tests may be run against both a replica set and shard cluster. @@ -106,155 +106,160 @@ Drivers should test that transactions IDs are always included in commands for su The following tests ensure that retryable writes work properly with replica sets and sharded clusters. -- Test that retryable writes raise an exception when using the MMAPv1 storage engine. For this test, execute a write - operation, such as `insertOne`, which should generate an exception. Assert that the error message is the replacement - error message: +### 1. Test that retryable writes raise an exception when using the MMAPv1 storage engine. - ``` - This MongoDB deployment does not support retryable writes. Please add - retryWrites=false to your connection string. - ``` +For this test, execute a write operation, such as `insertOne`, which should generate an exception. Assert that the error +message is the replacement error message: - and the error code is 20. +``` +This MongoDB deployment does not support retryable writes. Please add +retryWrites=false to your connection string. +``` - > [!NOTE] - > Drivers that rely on `serverStatus` to determine the storage engine in use MAY skip this test for sharded clusters, - > since `mongos` does not report this information in its `serverStatus` response. +and the error code is 20. -- Test that drivers properly retry after encountering PoolClearedErrors. This test MUST be implemented by any driver - that implements the CMAP specification. +> [!NOTE] +> Drivers that rely on `serverStatus` to determine the storage engine in use MAY skip this test for sharded clusters, +> since `mongos` does not report this information in its `serverStatus` response. - This test requires MongoDB 4.3.4+ for both the `errorLabels` and `blockConnection` fail point options. +### 2. Test that drivers properly retry after encountering PoolClearedErrors. - - Create a client with maxPoolSize=1 and retryWrites=true. If testing against a sharded deployment, be sure to connect - to only a single mongos. +This test MUST be implemented by any driver that implements the CMAP specification. - - Enable the following failpoint: +This test requires MongoDB 4.3.4+ for both the `errorLabels` and `blockConnection` fail point options. - ```javascript - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 91, - blockConnection: true, - blockTimeMS: 1000, - errorLabels: ["RetryableWriteError"] - } +1. Create a client with maxPoolSize=1 and retryWrites=true. If testing against a sharded deployment, be sure to connect + to only a single mongos. + +2. Enable the following failpoint: + +```javascript +{ + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 91, + blockConnection: true, + blockTimeMS: 1000, + errorLabels: ["RetryableWriteError"] } - ``` +} +``` - - Start two threads and attempt to perform an `insertOne` simultaneously on both. +3. Start two threads and attempt to perform an `insertOne` simultaneously on both. - - Verify that both `insertOne` attempts succeed. +4. Verify that both `insertOne` attempts succeed. - - Via CMAP monitoring, assert that the first check out succeeds. +5. Via CMAP monitoring, assert that the first check out succeeds. - - Via CMAP monitoring, assert that a PoolClearedEvent is then emitted. +6. Via CMAP monitoring, assert that a PoolClearedEvent is then emitted. - - Via CMAP monitoring, assert that the second check out then fails due to a connection error. +7. Via CMAP monitoring, assert that the second check out then fails due to a connection error. - - Via Command Monitoring, assert that exactly three `insert` CommandStartedEvents were observed in total. +8. Via Command Monitoring, assert that exactly three `insert` CommandStartedEvents were observed in total. - - Disable the failpoint. +9. Disable the failpoint. -- Test that drivers return the original error after encountering a WriteConcernError with a RetryableWriteError label. - This test MUST +### 3. Test that drivers return the original error after encountering a WriteConcernError with a RetryableWriteError label. - - be implemented by any driver that implements the Command Monitoring specification, +This test MUST: - - only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. +1. be implemented by any driver that implements the Command Monitoring specification, - - be run against server versions 6.0 and above. +2. only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. - Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the - subsequent retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or - write a unit test to cover the same sequence of events. +3. be run against server versions 6.0 and above. - - Create a client with `retryWrites=true`. +``` +Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the +subsequent retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or +write a unit test to cover the same sequence of events. +``` - - Configure a fail point with error code `91` (ShutdownInProgress): +4. Create a client with `retryWrites=true`. - ```javascript - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorLabels: ["RetryableWriteError"], - writeConcernError: { code: 91 } - } +5. Configure a fail point with error code `91` (ShutdownInProgress): + +```javascript +{ + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorLabels: ["RetryableWriteError"], + writeConcernError: { code: 91 } } - ``` - - - Via the command monitoring CommandSucceededEvent, configure a fail point with error code `10107` - (NotWritablePrimary) and a NoWritesPerformed label: - - ```javascript - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorCode: 10107, - errorLabels: ["RetryableWriteError", "NoWritesPerformed"] - } +} +``` + +6. Via the command monitoring CommandSucceededEvent, configure a fail point with error code `10107` (NotWritablePrimary) + and a NoWritesPerformed label: + +```javascript +{ + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorCode: 10107, + errorLabels: ["RetryableWriteError", "NoWritesPerformed"] } - ``` - - Drivers SHOULD only configure the `10107` fail point command if the the succeeded event is for the `91` error - configured in step 2. - - - Attempt an `insertOne` operation on any record for any database and collection. For the resulting error, assert that - the associated error code is `91`. - - - Disable the fail point: - - ```javascript - { - configureFailPoint: "failCommand", - mode: "off" - } - ``` - -- Test that in a sharded cluster writes are retried on a different mongos when one is available. This test MUST be - executed against a sharded cluster that has at least two mongos instances, supports `retryWrites=true`, has enabled - the `configureFailPoint` command, and supports the `errorLabels` field (MongoDB 4.3.1+). - - Note: this test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior - intended to be tested) from "retry on a different mongos due to normal SDAM randomized suitable server selection". - Verify relevant code paths are correctly executed by the tests using external means such as a logging, debugger, code - coverage tool, etc. - - - Create two clients `s0` and `s1` that each connect to a single mongos from the sharded cluster. They must not - connect to the same mongos. - - - Configure the following fail point for both `s0` and `s1`: - - ```javascript - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } +} +``` + +Drivers SHOULD only configure the `10107` fail point command if the the succeeded event is for the `91` error configured +in step 2. + +7. Attempt an `insertOne` operation on any record for any database and collection. For the resulting error, assert that + the associated error code is `91`. + +8. Disable the fail point: + +```javascript +{ + configureFailPoint: "failCommand", + mode: "off" +} +``` + +### 4. Test that in a sharded cluster writes are retried on a different mongos when one is available. + +This test MUST be executed against a sharded cluster that has at least two mongos instances, supports +`retryWrites=true`, has enabled the `configureFailPoint` command, and supports the `errorLabels` field (MongoDB 4.3.1+). + +> [!NOTE] This test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior +> intended to be tested) from "retry on a different mongos due to normal SDAM randomized suitable server selection". +> Verify relevant code paths are correctly executed by the tests using external means such as a logging, debugger, code +> coverage tool, etc. + +1. Create two clients `s0` and `s1` that each connect to a single mongos from the sharded cluster. They must not connect + to the same mongos. + +2. Configure the following fail point for both `s0` and `s1`: + +```javascript +{ + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 6, + errorLabels: ["RetryableWriteError"] } - ``` +} +``` - - Create a client `client` with `retryWrites=true` that connects to the cluster using the same two mongoses as `s0` - and `s1`. +3. Create a client `client` with `retryWrites=true` that connects to the cluster using the same two mongoses as `s0` and + `s1`. - - Enable failed command event monitoring for `client`. +4. Enable failed command event monitoring for `client`. - - Execute an `insert` command with `client`. Assert that the command failed. +5. Execute an `insert` command with `client`. Assert that the command failed. - - Assert that two failed command events occurred. Assert that the failed command events occurred on different - mongoses. +6. Assert that two failed command events occurred. Assert that the failed command events occurred on different mongoses. - - Disable the fail points on both `s0` and `s1`. +7. Disable the fail points on both `s0` and `s1`. ## Changelog From 98455a784ebba0c384f63bdc27b9d766179fb23e Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:33:37 -0500 Subject: [PATCH 03/18] rendering --- source/retryable-writes/tests/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 56927de35e..102c570f94 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -1,7 +1,5 @@ # Retryable Write Tests -______________________________________________________________________ - ## Introduction The YAML and JSON files in this directory are platform-independent tests meant to exercise a driver's implementation of From 152a4ca98840f42a5a66b01841449d2bb3f37034 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:35:39 -0500 Subject: [PATCH 04/18] update pre-commit version --- .github/workflows/lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 2d3490a0da..e16543d331 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -16,7 +16,7 @@ jobs: - uses: actions/setup-python@v4 # ref: https://github.com/pre-commit/action - - uses: pre-commit/action@v3.0.0 + - uses: pre-commit/action@v3.7.0 - name: Help message if pre-commit fail if: ${{ failure() }} run: | From d4f5509231b512a9fcd9ab61f6aa0f85d05ebf5d Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:36:35 -0500 Subject: [PATCH 05/18] update pre-commit version --- .github/workflows/lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index e16543d331..6d95686c86 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -16,7 +16,7 @@ jobs: - uses: actions/setup-python@v4 # ref: https://github.com/pre-commit/action - - uses: pre-commit/action@v3.7.0 + - uses: pre-commit/action@v3.0.1 - name: Help message if pre-commit fail if: ${{ failure() }} run: | From c4ffee9d1fe3403ec07a2f40331d366f5d0916d1 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:39:25 -0500 Subject: [PATCH 06/18] update pre-commit --- .github/workflows/lint.yml | 17 ++++------------- .pre-commit-config.yaml | 26 +++++++++++++------------- 2 files changed, 17 insertions(+), 26 deletions(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 6d95686c86..3fd262d089 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -14,16 +14,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v4 - - # ref: https://github.com/pre-commit/action - - uses: pre-commit/action@v3.0.1 - - name: Help message if pre-commit fail - if: ${{ failure() }} - run: | - echo "You can install pre-commit hooks to automatically run formatting" - echo "on each commit with:" - echo " pre-commit install" - echo "or you can run by hand on staged files with" - echo " pre-commit run" - echo "or after-the-fact on already committed files with" - echo " pre-commit run --all-files --hook-stage manual" + - name: "Run pre-commit" + run: + pip install -U pre-commit + pre-commit run --all-files --hook-stage manual diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 92862432fd..cd78710252 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,7 +1,7 @@ repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.5.0 + rev: v4.6.0 hooks: - id: check-case-conflict - id: check-executables-have-shebangs @@ -12,18 +12,10 @@ repos: #- id: trailing-whitespace #- id: check-json -- repo: https://github.com/executablebooks/mdformat - rev: 0.7.17 - hooks: - - id: mdformat - args: ["--wrap=120", "--number"] - additional_dependencies: - [mdformat-gfm, mdformat-frontmatter, mdformat-footnote, mdformat-gfm-alerts] - # We use the Python version instead of the original version which seems to require Docker # https://github.com/koalaman/shellcheck-precommit - repo: https://github.com/shellcheck-py/shellcheck-py - rev: v0.9.0.6 + rev: v0.10.0.1 hooks: - id: shellcheck name: shellcheck @@ -41,9 +33,17 @@ repos: types: [markdown] entry: bash scripts/check_anchors/check-anchors.sh language: node + +- repo: https://github.com/executablebooks/mdformat + rev: 0.7.17 + hooks: + - id: mdformat + args: ["--wrap=120", "--number"] + additional_dependencies: + [mdformat-gfm, mdformat-frontmatter, mdformat-footnote, mdformat-gfm-alerts] - repo: https://github.com/tcort/markdown-link-check - rev: v3.11.2 + rev: v3.12.2 hooks: - id: markdown-link-check args: ["-c", "markdown_link_config.json"] @@ -57,7 +57,7 @@ repos: stages: [manual] - repo: https://github.com/python-jsonschema/check-jsonschema - rev: 0.27.3 + rev: 0.28.4 hooks: - id: check-github-workflows @@ -69,7 +69,7 @@ repos: - id: rst-inline-touching-normal - repo: https://github.com/codespell-project/codespell - rev: "v2.2.6" + rev: "v2.3.0" hooks: - id: codespell args: ["-L", "fle,re-use,merchantibility,synching,crate,nin,infinit,te"] From f243b1d8cda0213cb26e8e8b775638dac4f03771 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:43:26 -0500 Subject: [PATCH 07/18] fix action --- .github/workflows/lint.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 3fd262d089..02f52237e2 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -15,6 +15,6 @@ jobs: - uses: actions/checkout@v4 - uses: actions/setup-python@v4 - name: "Run pre-commit" - run: - pip install -U pre-commit + run: | + pip install -U -q pre-commit pre-commit run --all-files --hook-stage manual From 68db81961c845dcb884428e69278c3dcb8e126b1 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:47:25 -0500 Subject: [PATCH 08/18] try again --- .pre-commit-config.yaml | 2 +- source/server-discovery-and-monitoring/server-monitoring.md | 2 +- source/transactions/transactions.md | 2 +- source/unified-test-format/unified-test-format.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index cd78710252..5355d35978 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -72,7 +72,7 @@ repos: rev: "v2.3.0" hooks: - id: codespell - args: ["-L", "fle,re-use,merchantibility,synching,crate,nin,infinit,te"] + args: ["-L", "fle,re-use,merchantibility,synching,crate,nin,infinit,te,checkin"] exclude: | (?x)^(.*\.rst )$ diff --git a/source/server-discovery-and-monitoring/server-monitoring.md b/source/server-discovery-and-monitoring/server-monitoring.md index 2e30583a4d..640685f2aa 100644 --- a/source/server-discovery-and-monitoring/server-monitoring.md +++ b/source/server-discovery-and-monitoring/server-monitoring.md @@ -581,7 +581,7 @@ class Monitor(Thread): wait() def setUpConnection(): - # Take the mutex to avoid a data race becauase this code writes to the connection field and a concurrent + # Take the mutex to avoid a data race because this code writes to the connection field and a concurrent # cancelCheck call could be reading from it. with lock: # Server API versioning implies that the server supports hello. diff --git a/source/transactions/transactions.md b/source/transactions/transactions.md index 5c922eee58..0ad7dd559c 100644 --- a/source/transactions/transactions.md +++ b/source/transactions/transactions.md @@ -636,7 +636,7 @@ Drivers MUST unpin a ClientSession in the following situations: 1. The transaction is aborted. The session MUST be unpinned regardless of whether or the `abortTransaction` command succeeds or fails, or was executed at all. If the operation fails with a retryable error, the session MUST be unpinned before performing server selection for the retry. -2. Any operation in the transcation, including `commitTransaction` fails with a TransientTransactionError. Transient +2. Any operation in the transaction, including `commitTransaction` fails with a TransientTransactionError. Transient errors indicate that the transaction in question has already been aborted or that the pinnned mongos is down/unavailable. Unpinning the session ensures that a subsequent `abortTransaction` (or `commitTransaction`) does not block waiting on a server that is unreachable. diff --git a/source/unified-test-format/unified-test-format.md b/source/unified-test-format/unified-test-format.md index 689e281353..883c783fab 100644 --- a/source/unified-test-format/unified-test-format.md +++ b/source/unified-test-format/unified-test-format.md @@ -834,7 +834,7 @@ The structure of this object is as follows: - `ignoreResultAndError`: Optional boolean. If true, both the error and result for the operation MUST be ignored. - This field is mutally exclusive with [expectResult](#operation_expectResult), [expectError](#operation_expectError), + This field is mutually exclusive with [expectResult](#operation_expectResult), [expectError](#operation_expectError), and [saveResultAsEntity](#operation_saveResultAsEntity). This field SHOULD NOT be used for [Special Test Operations](#special-test-operations) (i.e. `object: testRunner`). From 6bdaaadb6c8a4723962b14f13f4a19d23f93585a Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:48:38 -0500 Subject: [PATCH 09/18] try again --- source/retryable-writes/tests/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 102c570f94..3148dd0bd8 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -272,13 +272,15 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2024-01-05: Fix typo in prose test title. - 2024-01-03: Note server version requirements for fail point options and revise\ - tests to specify the `errorLabels` + tests to specify the + `errorLabels`\ option at the top-level instead of within `writeConcernError`. - 2023-08-26: Add prose tests for retrying in a sharded cluster. - 2022-08-30: Add prose test verifying correct error handling for errors with\ - the NoWritesPerformed label, which is to + the NoWritesPerformed label, which is + to\ return the original error. - 2022-04-22: Clarifications to `serverless` and `useMultipleMongoses`. @@ -298,8 +300,10 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2019-06-07: Mention $merge stage for aggregate alongside $out - 2019-03-01: Add top-level `runOn` field to denote server version and/or\ - topology requirements requirements for the - test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now expressed within + topology requirements requirements for + the\ + test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now expressed + within\ `runOn` elements. Add test-level `useMultipleMongoses` field. From ea17642f19154f7254beec703773dcd6bf75cee1 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 16:53:04 -0500 Subject: [PATCH 10/18] try again --- source/retryable-writes/tests/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 3148dd0bd8..bb68ff5c21 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -266,20 +266,23 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2024-02-27: Convert legacy retryable writes tests to unified format. - 2024-02-21: Update prose test 4 and 5 to workaround SDAM behavior preventing\ - execution of deprioritization code + execution of deprioritization + code\ paths. - 2024-01-05: Fix typo in prose test title. - 2024-01-03: Note server version requirements for fail point options and revise\ - tests to specify the + tests to specify + the\ `errorLabels`\ option at the top-level instead of within `writeConcernError`. - 2023-08-26: Add prose tests for retrying in a sharded cluster. - 2022-08-30: Add prose test verifying correct error handling for errors with\ - the NoWritesPerformed label, which is + the NoWritesPerformed label, which + is\ to\ return the original error. @@ -302,7 +305,8 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2019-03-01: Add top-level `runOn` field to denote server version and/or\ topology requirements requirements for the\ - test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now expressed + test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now + expressed\ within\ `runOn` elements. From a35e1d7f429a3f6eea979895acd14f50b51a4db0 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 17:00:16 -0500 Subject: [PATCH 11/18] try again --- source/retryable-writes/tests/README.md | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index bb68ff5c21..9488dbc373 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -226,10 +226,11 @@ in step 2. This test MUST be executed against a sharded cluster that has at least two mongos instances, supports `retryWrites=true`, has enabled the `configureFailPoint` command, and supports the `errorLabels` field (MongoDB 4.3.1+). -> [!NOTE] This test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior +> [!NOTE] +> This test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior > intended to be tested) from "retry on a different mongos due to normal SDAM randomized suitable server selection". > Verify relevant code paths are correctly executed by the tests using external means such as a logging, debugger, code -> coverage tool, etc. +> coverage tool, etc. --> --> 1. Create two clients `s0` and `s1` that each connect to a single mongos from the sharded cluster. They must not connect to the same mongos. @@ -266,24 +267,19 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2024-02-27: Convert legacy retryable writes tests to unified format. - 2024-02-21: Update prose test 4 and 5 to workaround SDAM behavior preventing\ - execution of deprioritization - code\ + execution of deprioritization code paths. - 2024-01-05: Fix typo in prose test title. - 2024-01-03: Note server version requirements for fail point options and revise\ - tests to specify - the\ - `errorLabels`\ + tests to specify the `errorLabels` option at the top-level instead of within `writeConcernError`. - 2023-08-26: Add prose tests for retrying in a sharded cluster. - 2022-08-30: Add prose test verifying correct error handling for errors with\ - the NoWritesPerformed label, which - is\ - to\ + the NoWritesPerformed label, which is to return the original error. - 2022-04-22: Clarifications to `serverless` and `useMultipleMongoses`. @@ -303,11 +299,8 @@ This test MUST be executed against a sharded cluster that has at least two mongo - 2019-06-07: Mention $merge stage for aggregate alongside $out - 2019-03-01: Add top-level `runOn` field to denote server version and/or\ - topology requirements requirements for - the\ - test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now - expressed\ - within\ + topology requirements requirements for the + test file. Removes the `minServerVersion` and `maxServerVersion` top-level fields, which are now expressed within `runOn` elements. Add test-level `useMultipleMongoses` field. From 5c24028f1143f9ff02b8f51dfb4784684824093f Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Thu, 30 May 2024 17:00:50 -0500 Subject: [PATCH 12/18] cleanup --- source/retryable-writes/tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 9488dbc373..253e02631e 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -230,7 +230,7 @@ This test MUST be executed against a sharded cluster that has at least two mongo > This test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior > intended to be tested) from "retry on a different mongos due to normal SDAM randomized suitable server selection". > Verify relevant code paths are correctly executed by the tests using external means such as a logging, debugger, code -> coverage tool, etc. --> --> +> coverage tool, etc. 1. Create two clients `s0` and `s1` that each connect to a single mongos from the sharded cluster. They must not connect to the same mongos. From c5d83ff0a92dcc559b6ed4367cd05fdccacd78cd Mon Sep 17 00:00:00 2001 From: Jeremy Mikola Date: Fri, 31 May 2024 10:36:28 -0400 Subject: [PATCH 13/18] Fix list indentation --- source/retryable-writes/tests/README.md | 132 ++++++++++++------------ 1 file changed, 64 insertions(+), 68 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 253e02631e..e97150669b 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -131,19 +131,19 @@ This test requires MongoDB 4.3.4+ for both the `errorLabels` and `blockConnectio 2. Enable the following failpoint: -```javascript -{ - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 91, - blockConnection: true, - blockTimeMS: 1000, - errorLabels: ["RetryableWriteError"] - } -} -``` + ```javascript + { + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 91, + blockConnection: true, + blockTimeMS: 1000, + errorLabels: ["RetryableWriteError"] + } + } + ``` 3. Start two threads and attempt to perform an `insertOne` simultaneously on both. @@ -163,63 +163,59 @@ This test requires MongoDB 4.3.4+ for both the `errorLabels` and `blockConnectio This test MUST: -1. be implemented by any driver that implements the Command Monitoring specification, - -2. only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. - -3. be run against server versions 6.0 and above. +- be implemented by any driver that implements the Command Monitoring specification, +- only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. +- be run against server versions 6.0 and above. -``` Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the subsequent retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or write a unit test to cover the same sequence of events. -``` -4. Create a client with `retryWrites=true`. +1. Create a client with `retryWrites=true`. -5. Configure a fail point with error code `91` (ShutdownInProgress): +2. Configure a fail point with error code `91` (ShutdownInProgress): -```javascript -{ - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorLabels: ["RetryableWriteError"], - writeConcernError: { code: 91 } - } -} -``` + ```javascript + { + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorLabels: ["RetryableWriteError"], + writeConcernError: { code: 91 } + } + } + ``` -6. Via the command monitoring CommandSucceededEvent, configure a fail point with error code `10107` (NotWritablePrimary) +3. Via the command monitoring CommandSucceededEvent, configure a fail point with error code `10107` (NotWritablePrimary) and a NoWritesPerformed label: -```javascript -{ - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorCode: 10107, - errorLabels: ["RetryableWriteError", "NoWritesPerformed"] - } -} -``` - -Drivers SHOULD only configure the `10107` fail point command if the the succeeded event is for the `91` error configured -in step 2. - -7. Attempt an `insertOne` operation on any record for any database and collection. For the resulting error, assert that + ```javascript + { + configureFailPoint: "failCommand", + mode: {times: 1}, + data: { + failCommands: ["insert"], + errorCode: 10107, + errorLabels: ["RetryableWriteError", "NoWritesPerformed"] + } + } + ``` + + Drivers SHOULD only configure the `10107` fail point command if the the succeeded event is for the `91` error + configured in step 2. + +4. Attempt an `insertOne` operation on any record for any database and collection. For the resulting error, assert that the associated error code is `91`. -8. Disable the fail point: +5. Disable the fail point: -```javascript -{ - configureFailPoint: "failCommand", - mode: "off" -} -``` + ```javascript + { + configureFailPoint: "failCommand", + mode: "off" + } + ``` ### 4. Test that in a sharded cluster writes are retried on a different mongos when one is available. @@ -237,17 +233,17 @@ This test MUST be executed against a sharded cluster that has at least two mongo 2. Configure the following fail point for both `s0` and `s1`: -```javascript -{ - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } -} -``` + ```javascript + { + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 6, + errorLabels: ["RetryableWriteError"] + } + } + ``` 3. Create a client `client` with `retryWrites=true` that connects to the cluster using the same two mongoses as `s0` and `s1`. From 10a082a7171808001c3c5e3703212b60a1ce9bf1 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Fri, 31 May 2024 09:47:17 -0500 Subject: [PATCH 14/18] add missing test --- source/retryable-writes/tests/README.md | 37 +++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index e97150669b..b5a3b74155 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -167,9 +167,9 @@ This test MUST: - only run against replica sets as mongos does not propagate the NoWritesPerformed label to the drivers. - be run against server versions 6.0 and above. -Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the -subsequent retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or -write a unit test to cover the same sequence of events. +Additionally, this test requires drivers to set a fail point after an `insertOne` operation but before the subsequent +retry. Drivers that are unable to set a failCommand after the CommandSucceededEvent SHOULD use mocking or write a unit +test to cover the same sequence of events. 1. Create a client with `retryWrites=true`. @@ -256,6 +256,37 @@ This test MUST be executed against a sharded cluster that has at least two mongo 7. Disable the fail points on both `s0` and `s1`. +### 5. Test that in a sharded cluster on the same mongos if no other is available + +This test MUST be executed against a sharded cluster + +1. Ensure that a test is run against a sharded cluster. If there are multiple mongoses in the cluster, pick one to test + against. + +2. Create a client that connects to the mongos using the direct connection, and configure the following fail point on + the mongos:: + +```javascript +{ + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 6, + errorLabels: ["RetryableWriteError"], + closeConnection: true + } +} +``` + +3. Create a client with `retryWrites=true` that connects to the cluster, providing the selected mongos as the seed. + +4. Enable command monitoring, and execute a write command that is supposed to fail. + +5. Asserts that there was a failed command and a successful command event. + +6. Disable the fail point. + ## Changelog - 2024-05-30: Migrated from reStructuredText to Markdown. From 3407e6c9c31f70f30ed738f649ad5a11479bb245 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Fri, 31 May 2024 09:48:24 -0500 Subject: [PATCH 15/18] add indent --- source/retryable-writes/tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index b5a3b74155..73fe86054a 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -264,7 +264,7 @@ This test MUST be executed against a sharded cluster against. 2. Create a client that connects to the mongos using the direct connection, and configure the following fail point on - the mongos:: + the mongos: ```javascript { From 1ee22b41255ed10860dad8eccad4849c811e3e3f Mon Sep 17 00:00:00 2001 From: Jeremy Mikola Date: Fri, 31 May 2024 11:47:00 -0400 Subject: [PATCH 16/18] Fix code block indentation --- source/retryable-writes/tests/README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 73fe86054a..ca5afca40e 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -266,18 +266,18 @@ This test MUST be executed against a sharded cluster 2. Create a client that connects to the mongos using the direct connection, and configure the following fail point on the mongos: -```javascript -{ - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"], - closeConnection: true - } -} -``` + ```javascript + { + configureFailPoint: "failCommand", + mode: { times: 1 }, + data: { + failCommands: ["insert"], + errorCode: 6, + errorLabels: ["RetryableWriteError"], + closeConnection: true + } + } + ``` 3. Create a client with `retryWrites=true` that connects to the cluster, providing the selected mongos as the seed. From 4c35c2ab656ab2fd239942626891ceef91221793 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Fri, 31 May 2024 11:03:40 -0500 Subject: [PATCH 17/18] update test --- source/retryable-writes/tests/README.md | 26 +++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index ca5afca40e..9f18353225 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -256,15 +256,17 @@ This test MUST be executed against a sharded cluster that has at least two mongo 7. Disable the fail points on both `s0` and `s1`. -### 5. Test that in a sharded cluster on the same mongos if no other is available +### 5. Test that in a sharded cluster writes are retried on the same mongos when no others are available. -This test MUST be executed against a sharded cluster +This test MUST be executed against a sharded cluster that supports `retryWrites=true`, has enabled the +`configureFailPoint` command, and supports the `errorLabels` field (MongoDB 4.3.1+). -1. Ensure that a test is run against a sharded cluster. If there are multiple mongoses in the cluster, pick one to test - against. +Note: this test cannot reliably distinguish "retry on a different mongos due to server deprioritization" (the behavior +intended to be tested) from "retry on a different mongos due to normal SDAM behavior of randomized suitable server +selection". Verify relevant code paths are correctly executed by the tests using external means such as a logging, +debugger, code coverage tool, etc. -2. Create a client that connects to the mongos using the direct connection, and configure the following fail point on - the mongos: +1. Create a client `s0` that connects to a single mongos from the cluster. ```javascript { @@ -279,13 +281,17 @@ This test MUST be executed against a sharded cluster } ``` -3. Create a client with `retryWrites=true` that connects to the cluster, providing the selected mongos as the seed. +2. Create a client `client` with `directConnection=false` (when not set by default) and `retryWrites=true` that connects + to the cluster using the same single mongos as `s0`. -4. Enable command monitoring, and execute a write command that is supposed to fail. +3. Enable succeeded and failed command event monitoring for `client`. -5. Asserts that there was a failed command and a successful command event. +4. Execute an `insert` command with `client`. Assert that the command succeeded. -6. Disable the fail point. +5. Assert that exactly one failed command event and one succeeded command event occurred. Assert that both events + occurred on the same mongos. + +6. Disable the fail point on `s0`. ## Changelog From 839f0cfaa4f4ef0a2249d21a8f4907f2c611be7b Mon Sep 17 00:00:00 2001 From: Jeremy Mikola Date: Fri, 31 May 2024 12:18:02 -0400 Subject: [PATCH 18/18] Adding miss step to fifth prose test --- source/retryable-writes/tests/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index 9f18353225..e883ca368d 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -268,6 +268,8 @@ debugger, code coverage tool, etc. 1. Create a client `s0` that connects to a single mongos from the cluster. +2. Configure the following fail point for `s0`: + ```javascript { configureFailPoint: "failCommand", @@ -281,17 +283,17 @@ debugger, code coverage tool, etc. } ``` -2. Create a client `client` with `directConnection=false` (when not set by default) and `retryWrites=true` that connects +3. Create a client `client` with `directConnection=false` (when not set by default) and `retryWrites=true` that connects to the cluster using the same single mongos as `s0`. -3. Enable succeeded and failed command event monitoring for `client`. +4. Enable succeeded and failed command event monitoring for `client`. -4. Execute an `insert` command with `client`. Assert that the command succeeded. +5. Execute an `insert` command with `client`. Assert that the command succeeded. -5. Assert that exactly one failed command event and one succeeded command event occurred. Assert that both events +6. Assert that exactly one failed command event and one succeeded command event occurred. Assert that both events occurred on the same mongos. -6. Disable the fail point on `s0`. +7. Disable the fail point on `s0`. ## Changelog