diff --git a/source/index.md b/source/index.md index eaedba41dc..47298cf74c 100644 --- a/source/index.md +++ b/source/index.md @@ -25,6 +25,7 @@ - [MongoDB Driver Performance Benchmarking](benchmarking/benchmarking.md) - [OP_MSG](message/OP_MSG.md) - [Retryable Reads](retryable-reads/retryable-reads.md) +- [Retryable Writes](retryable-writes/retryable-writes.md) - [Server Selection](server-selection/server-selection.md) - [Server Selection Test Plan](server-selection/server-selection-tests.md) - [Unified Test Format](unified-test-format/unified-test-format.md) diff --git a/source/retryable-reads/retryable-reads.md b/source/retryable-reads/retryable-reads.md index eb79d181c7..05284e0e61 100644 --- a/source/retryable-reads/retryable-reads.md +++ b/source/retryable-reads/retryable-reads.md @@ -27,7 +27,7 @@ The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SH #### Retryable Error An error is considered retryable if it meets any of the criteria defined under -[Retryable Writes: Terms: Retryable Error](../retryable-writes/retryable-writes.rst#terms), minus the final criterion +[Retryable Writes: Terms: Retryable Error](../retryable-writes/retryable-writes.md#terms), minus the final criterion about write concern errors. For convenience, the relevant criteria have been adapted to retryable reads and reproduced below. @@ -64,14 +64,14 @@ This specification introduces the following client-level configuration option. This boolean option determines whether retryable behavior will be applied to all read operations executed within the MongoClient. This option MUST default to true. -[As with retryable writes](../retryable-writes/retryable-writes.rst#retrywrites), this option MUST NOT be configurable -at the level of an individual read operation, collection object, or database object. Drivers that expose a "high" and +[As with retryable writes](../retryable-writes/retryable-writes.md#retrywrites), this option MUST NOT be configurable at +the level of an individual read operation, collection object, or database object. Drivers that expose a "high" and "core" API (e.g. Java and C# driver) MUST NOT expose a configurable option at the level of an individual read operation, collection object, or database object in "high", but MAY expose the option in "core." ##### Naming Deviations -[As with retryable writes](../retryable-writes/retryable-writes.rst#retrywrites), drivers MUST use the defined name of +[As with retryable writes](../retryable-writes/retryable-writes.md#retrywrites), drivers MUST use the defined name of `retryReads` for the connection string parameter to ensure portability of connection strings across applications and drivers. If drivers solicit MongoClient options through another mechanism (e.g. an options dictionary provided to the MongoClient constructor), drivers SHOULD use the defined name but MAY deviate to comply with their existing conventions. @@ -165,7 +165,7 @@ Drivers MUST NOT retry the following operations: #### Executing Retryable Read Commands Executing retryable read commands is extremely similar to -[executing retryable write commands](../retryable-writes/retryable-writes.rst#executing-retryable-write-commands). The +[executing retryable write commands](../retryable-writes/retryable-writes.md#executing-retryable-write-commands). The following explanation for executing retryable read commands has been adapted from the explanation for executing retryable write commands. @@ -264,7 +264,7 @@ When retrying a read command, drivers MUST NOT resend the original wire protocol #### Pseudocode The following pseudocode for executing retryable read commands has been adapted from -[the pseudocode for executing retryable write commands](../retryable-writes/retryable-writes.rst#executing-retryable-write-commands) +[the pseudocode for executing retryable write commands](../retryable-writes/retryable-writes.md#executing-retryable-write-commands) and reflects the flow described above. ```typescript @@ -382,12 +382,12 @@ function executeRetryableRead(command, session) { ### Logging Retry Attempts -[As with retryable writes](../retryable-writes/retryable-writes.rst#logging-retry-attempts), drivers MAY choose to log +[As with retryable writes](../retryable-writes/retryable-writes.md#logging-retry-attempts), drivers MAY choose to log retry attempts for read operations. This specification does not define a format for such log messages. ### Command Monitoring -[As with retryable writes](../retryable-writes/retryable-writes.rst#command-monitoring), in accordance with the +[As with retryable writes](../retryable-writes/retryable-writes.md#command-monitoring), in accordance with the [Command Logging and Monitoring](../command-logging-and-monitoring/command-logging-and-monitoring.rst) specification, drivers MUST guarantee that each `CommandStartedEvent` has either a correlating `CommandSucceededEvent` or `CommandFailedEvent` and that every "command started" log message has either a correlating "command succeeded" log @@ -424,7 +424,7 @@ to be extended to include support for retryable behavior for read operations. ## Design Rationale The design of this specification is based off the -[Retryable Writes specification](../retryable-writes/retryable-writes.rst#design-rationale). It modifies the driver API +[Retryable Writes specification](../retryable-writes/retryable-writes.md#design-rationale). It modifies the driver API as little as possible to introduce the concept retryable behavior for read operations. Alternative retry strategies (e.g. exponential back-off, incremental intervals, regular intervals, immediate retry, @@ -496,7 +496,7 @@ circumstances, so that we may revisit this decision to disallow trying `getMore( ### Why are read operations only retried once by default? -[Read operations are only retried once for the same reasons that writes are also only retried once.](../retryable-writes/retryable-writes.rst#why-are-write-operations-only-retried-once-by-default) +[Read operations are only retried once for the same reasons that writes are also only retried once.](../retryable-writes/retryable-writes.md#why-are-write-operations-only-retried-once-by-default) For convenience's sake, that reasoning has been adapted for reads and reproduced below: The spec concerns itself with retrying read operations that encounter a retryable error (i.e. no response due to network @@ -517,7 +517,7 @@ restarts during planned maintenance events. ### Can drivers resend the same wire protocol message on retry attempts? No. -[This is in contrast to the answer supplied in in the retryable writes specification.](../retryable-writes/retryable-writes.rst#can-drivers-resend-the-same-wire-protocol-message-on-retry-attempts) +[This is in contrast to the answer supplied in in the retryable writes specification.](../retryable-writes/retryable-writes.md#can-drivers-resend-the-same-wire-protocol-message-on-retry-attempts) However, when retryable writes were implemented, no driver actually chose to resend the same wire protocol message. Today, if a driver attempted to resend the same wire protocol message, this could violate [the rules for gossiping $clusterTime](../sessions/driver-sessions.rst#gossipping-the-cluster-time): specifically diff --git a/source/retryable-writes/retryable-writes.md b/source/retryable-writes/retryable-writes.md index f73e28a9dc..aa300b3455 100644 --- a/source/retryable-writes/retryable-writes.md +++ b/source/retryable-writes/retryable-writes.md @@ -1,928 +1,777 @@ -================ -Retryable Writes -================ - -:Status: Accepted -:Minimum Server Version: 3.6 - -.. contents:: - --------- - -Abstract -======== - -MongoDB 3.6 will implement support for server sessions, which are shared -resources within a cluster identified by a session ID. Drivers compatible with -MongoDB 3.6 will also implement support for client sessions, which are always -associated with a server session and will allow for certain commands to be -executed within the context of a server session. - -Additionally, MongoDB 3.6 will utilize server sessions to allow some write -commands to specify a transaction ID to enforce at-most-once semantics for the -write operation(s) and allow for retrying the operation if the driver fails to -obtain a write result (e.g. network error or "not writable primary" error after -a replica set failover). This specification will outline how an API for retryable -write operations will be implemented in drivers. The specification will define an -option to enable retryable writes for an application and describe how a -transaction ID will be provided to write commands executed therein. - -META -==== - -The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", -"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be -interpreted as described in `RFC 2119 `_. - -Specification -============= - -Terms ------ - -Transaction ID - The transaction ID identifies the transaction as part of which the command is - running. In a write command where the client has requested retryable - behavior, it is expressed by the top-level ``lsid`` and ``txnNumber`` fields. - The ``lsid`` component is the corresponding server session ID. which is a - BSON value defined in the `Driver Session`_ specification. The ``txnNumber`` - component is a monotonically increasing (per server session), positive 64-bit - integer. - -.. _Driver Session: ../sessions/driver-sessions.rst - -ClientSession - Driver object representing a client session, which is defined in the - `Driver Session`_ specification. This object is always associated with a - server session; however, drivers will pool server sessions so that creating a - ClientSession will not always entail creation of a new server session. The - name of this object MAY vary across drivers. - -Retryable Error - An error is considered retryable if it has a RetryableWriteError label in - its top-level "errorLabels" field. See `Determining Retryable Errors`_ for - more information. - -Additional terms may be defined in the `Driver Session`_ specification. - -Naming Deviations ------------------ - -This specification defines the name for a new MongoClient option, -``retryWrites``. Drivers MUST use the defined name for the connection string -parameter to ensure portability of connection strings across applications and -drivers. +# Retryable Writes + +- Status: Accepted +- Minimum Server Version: 3.6 + +______________________________________________________________________ + +## Abstract + +MongoDB 3.6 will implement support for server sessions, which are shared resources within a cluster identified by a +session ID. Drivers compatible with MongoDB 3.6 will also implement support for client sessions, which are always +associated with a server session and will allow for certain commands to be executed within the context of a server +session. + +Additionally, MongoDB 3.6 will utilize server sessions to allow some write commands to specify a transaction ID to +enforce at-most-once semantics for the write operation(s) and allow for retrying the operation if the driver fails to +obtain a write result (e.g. network error or "not writable primary" error after a replica set failover). This +specification will outline how an API for retryable write operations will be implemented in drivers. The specification +will define an option to enable retryable writes for an application and describe how a transaction ID will be provided +to write commands executed therein. + +## META + +The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + +## Specification -If drivers solicit MongoClient options through another mechanism (e.g. options -dictionary provided to the MongoClient constructor), drivers SHOULD use the -defined name but MAY deviate to comply with their existing conventions. For -example, a driver may use ``retry_writes`` instead of ``retryWrites``. +### Terms -For any other names in the spec, drivers SHOULD use the defined name but MAY -deviate to comply with their existing conventions. +**Transaction ID**\ +The transaction ID identifies the transaction as part of which the command is running. In a write +command where the client has requested retryable behavior, it is expressed by the top-level `lsid` and `txnNumber` +fields. The `lsid` component is the corresponding server session ID. which is a BSON value defined in the +[Driver Session](../sessions/driver-sessions.rst) specification. The `txnNumber` component is a monotonically increasing +(per server session), positive 64-bit integer. -MongoClient Configuration -------------------------- +**ClientSession**\ +Driver object representing a client session, which is defined in the +[Driver Session](../sessions/driver-sessions.rst) specification. This object is always associated with a server session; +however, drivers will pool server sessions so that creating a ClientSession will not always entail creation of a new +server session. The name of this object MAY vary across drivers. + +**Retryable Error**\ +An error is considered retryable if it has a RetryableWriteError label in its top-level +"errorLabels" field. See [Determining Retryable Errors](#determining-retryable-errors) for more information. + +Additional terms may be defined in the [Driver Session](../sessions/driver-sessions.rst) specification. + +### Naming Deviations + +This specification defines the name for a new MongoClient option, `retryWrites`. Drivers MUST use the defined name for +the connection string parameter to ensure portability of connection strings across applications and drivers. + +If drivers solicit MongoClient options through another mechanism (e.g. options dictionary provided to the MongoClient +constructor), drivers SHOULD use the defined name but MAY deviate to comply with their existing conventions. For +example, a driver may use `retry_writes` instead of `retryWrites`. + +For any other names in the spec, drivers SHOULD use the defined name but MAY deviate to comply with their existing +conventions. + +### MongoClient Configuration This specification introduces the following client-level configuration option. -retryWrites -~~~~~~~~~~~ +#### retryWrites -This boolean option determines whether retryable behavior will be applied to all -supported write operations executed within the MongoClient. This option MUST -default to true. +This boolean option determines whether retryable behavior will be applied to all supported write operations executed +within the MongoClient. This option MUST default to true. -This option MUST NOT be configurable at the level of a database object, -collection object, or at the level of an individual write operation. +This option MUST NOT be configurable at the level of a database object, collection object, or at the level of an +individual write operation. -Requirements for Retryable Writes ---------------------------------- +### Requirements for Retryable Writes -Supported Server Versions -~~~~~~~~~~~~~~~~~~~~~~~~~ +#### Supported Server Versions -Like sessions, retryable writes require a MongoDB 3.6 replica set or shard -cluster operating with feature compatibility version 3.6 (i.e. the -``{setFeatureCompatibilityVersion: 3.6}`` administrative command has been run on -the cluster). Drivers MUST verify server eligibility by ensuring that -``maxWireVersion`` is at least six, the ``logicalSessionTimeoutMinutes`` -field is present in the server's ``hello`` or legacy hello response, and the server -type is not standalone. +Like sessions, retryable writes require a MongoDB 3.6 replica set or shard cluster operating with feature compatibility +version 3.6 (i.e. the `{setFeatureCompatibilityVersion: 3.6}` administrative command has been run on the cluster). +Drivers MUST verify server eligibility by ensuring that `maxWireVersion` is at least six, the +`logicalSessionTimeoutMinutes` field is present in the server's `hello` or legacy hello response, and the server type is +not standalone. -Retryable writes are only supported by storage engines that support document- -level locking. Notably, that excludes the MMAPv1 storage engine which is -available in both MongoDB 3.6 and 4.0. Since ``retryWrites`` defaults to -``true``, Drivers MUST raise an actionable error message when the server -returns code 20 with errmsg starting with "Transaction numbers". The -replacement error message MUST be:: +Retryable writes are only supported by storage engines that support document-level locking. Notably, that excludes the +MMAPv1 storage engine which is available in both MongoDB 3.6 and 4.0. Since `retryWrites` defaults to `true`, Drivers +MUST raise an actionable error message when the server returns code 20 with errmsg starting with "Transaction numbers". +The replacement error message MUST be: - This MongoDB deployment does not support retryable writes. Please add - retryWrites=false to your connection string. +``` +This MongoDB deployment does not support retryable writes. Please add +retryWrites=false to your connection string. +``` -If the server selected for the first attempt of a retryable write operation does -not support retryable writes, drivers MUST execute the write as if retryable -writes were not enabled. Drivers MUST NOT include a transaction ID in the write +If the server selected for the first attempt of a retryable write operation does not support retryable writes, drivers +MUST execute the write as if retryable writes were not enabled. Drivers MUST NOT include a transaction ID in the write command and MUST not retry the command under any circumstances. -In a sharded cluster, it is possible that mongos may appear to support retryable -writes but one or more shards in the cluster do not (e.g. replica set shard is -configured with feature compatibility version 3.4, a standalone is added as a -new shard). In these rare cases, a write command that fans out to a shard that -does not support retryable writes may partially fail and an error may be -reported in the write result from mongos (e.g. ``writeErrors`` array in the bulk -write result). This does not constitute a retryable error. Drivers MUST relay -such errors to the user. +In a sharded cluster, it is possible that mongos may appear to support retryable writes but one or more shards in the +cluster do not (e.g. replica set shard is configured with feature compatibility version 3.4, a standalone is added as a +new shard). In these rare cases, a write command that fans out to a shard that does not support retryable writes may +partially fail and an error may be reported in the write result from mongos (e.g. `writeErrors` array in the bulk write +result). This does not constitute a retryable error. Drivers MUST relay such errors to the user. -Supported Write Operations -~~~~~~~~~~~~~~~~~~~~~~~~~~ +#### Supported Write Operations MongoDB 3.6 will support retryability for some, but not all, write operations. -Supported single-statement write operations include ``insertOne()``, -``updateOne()``, ``replaceOne()``, ``deleteOne()``, ``findOneAndDelete()``, -``findOneAndReplace()``, and ``findOneAndUpdate()``. +Supported single-statement write operations include `insertOne()`, `updateOne()`, `replaceOne()`, `deleteOne()`, +`findOneAndDelete()`, `findOneAndReplace()`, and `findOneAndUpdate()`. -Supported multi-statement write operations include ``insertMany()`` and -``bulkWrite()``. The ordered option may be ``true`` or ``false``. In the case of -``bulkWrite()``, ``UpdateMany`` or ``DeleteMany`` operations within the -``requests`` parameter may make some write commands ineligible for retryability. -Drivers MUST evaluate eligibility for each write command sent as part of the -``bulkWrite()`` (after order and batch splitting) individually. Drivers MUST NOT -alter existing logic for order and batch splitting in an attempt to maximize -retryability for operations within a bulk write. +Supported multi-statement write operations include `insertMany()` and `bulkWrite()`. The ordered option may be `true` or +`false`. In the case of `bulkWrite()`, `UpdateMany` or `DeleteMany` operations within the `requests` parameter may make +some write commands ineligible for retryability. Drivers MUST evaluate eligibility for each write command sent as part +of the `bulkWrite()` (after order and batch splitting) individually. Drivers MUST NOT alter existing logic for order and +batch splitting in an attempt to maximize retryability for operations within a bulk write. -These methods above are defined in the `CRUD`_ specification. +These methods above are defined in the [CRUD](../crud/crud.md) specification. Later versions of MongoDB may add support for additional write operations. -Drivers MUST document operations that support retryable behavior and the -conditions for which retryability is determined (see: -`How will users know which operations are supported?`_). Drivers are not -required to exhaustively document all operations that do not support retryable -behavior. +Drivers MUST document operations that support retryable behavior and the conditions for which retryability is determined +(see: [How will users know which operations are supported?](#how-will-users-know-which-operations-are-supported)). +Drivers are not required to exhaustively document all operations that do not support retryable behavior. -Unsupported Write Operations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#### Unsupported Write Operations -Write commands specifying an unacknowledged write concern (e.g. ``{w: 0})``) do -not support retryable behavior. Drivers MUST NOT add a transaction ID to any -write command with an unacknowledged write concern executed within a MongoClient +Write commands specifying an unacknowledged write concern (e.g. `{w: 0})`) do not support retryable behavior. Drivers +MUST NOT add a transaction ID to any write command with an unacknowledged write concern executed within a MongoClient where retryable writes have been enabled. Drivers MUST NOT retry these commands. -Write commands where a single statement might affect multiple documents will not -be initially supported by MongoDB 3.6, although this may change in the future. -This includes an `update`_ command where any statement in the updates sequence -specifies a ``multi`` option of ``true`` or a `delete`_ command where any -statement in the ``deletes`` sequence specifies a ``limit`` option of ``0``. In -the context of the `CRUD`_ specification, this includes the ``updateMany()`` and -``deleteMany()`` methods and, in some cases, ``bulkWrite()``. Drivers MUST NOT -add a transaction ID to any single- or multi-statement write commands that -include one or more multi-document write operations. Drivers MUST NOT retry -these commands if they fail to return a response. With regard to -``bulkWrite()``, drivers MUST evaluate eligibility for each write command sent -as part of the ``bulkWrite()`` (after order and batch splitting) individually. - -.. _update: https://www.mongodb.com/docs/manual/reference/command/update/ -.. _delete: https://www.mongodb.com/docs/manual/reference/command/delete/ - -Write commands other than `insert`_, `update`_, `delete`_, or `findAndModify`_ -will not be initially supported by MongoDB 3.6, although this may change in the -future. This includes, but is not limited to, an `aggregate`_ command using a -write stage (e.g. ``$out``, ``$merge``). Drivers MUST NOT add a transaction ID -to these commands and MUST NOT retry these commands if they fail to return a -response. - -.. _insert: https://www.mongodb.com/docs/manual/reference/command/insert/ -.. _findAndModify: https://www.mongodb.com/docs/manual/reference/command/findAndModify/ -.. _aggregate: https://www.mongodb.com/docs/manual/reference/command/aggregate/ - -Retryable Writes Within Transactions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In MongoDB 4.0 the only supported retryable write commands within a transaction -are ``commitTransaction`` and ``abortTransaction``. Therefore drivers MUST NOT -retry write commands within transactions even when ``retryWrites`` has been -set to true on the ``MongoClient``. In addition, drivers MUST NOT add -the ``RetryableWriteError`` label to any error that occurs during a write -command within a transaction (excepting ``commitTransation`` -and ``abortTransaction``), even when ``retryWrites`` has been set to true on -the ``MongoClient``. - -Implementing Retryable Writes ------------------------------ - -Determining Retryable Errors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When connected to a MongoDB instance that supports retryable writes (versions 3.6+), -the driver MUST treat all errors with the RetryableWriteError label as retryable. -This error label can be found in the top-level "errorLabels" field of the error. - -RetryableWriteError Labels -^^^^^^^^^^^^^^^^^^^^^^^^^^ +Write commands where a single statement might affect multiple documents will not be initially supported by MongoDB 3.6, +although this may change in the future. This includes an +[update](https://www.mongodb.com/docs/manual/reference/command/update/) command where any statement in the updates +sequence specifies a `multi` option of `true` or a +[delete](https://www.mongodb.com/docs/manual/reference/command/delete/) command where any statement in the `deletes` +sequence specifies a `limit` option of `0`. In the context of the [CRUD](../crud/crud.md) specification, this includes +the `updateMany()` and `deleteMany()` methods and, in some cases, `bulkWrite()`. Drivers MUST NOT add a transaction ID +to any single- or multi-statement write commands that include one or more multi-document write operations. Drivers MUST +NOT retry these commands if they fail to return a response. With regard to `bulkWrite()`, drivers MUST evaluate +eligibility for each write command sent as part of the `bulkWrite()` (after order and batch splitting) individually. + +Write commands other than [insert](https://www.mongodb.com/docs/manual/reference/command/insert/), +[update](https://www.mongodb.com/docs/manual/reference/command/update/), +[delete](https://www.mongodb.com/docs/manual/reference/command/delete/), or +[findAndModify](https://www.mongodb.com/docs/manual/reference/command/findAndModify/) will not be initially supported by +MongoDB 3.6, although this may change in the future. This includes, but is not limited to, an +[aggregate](https://www.mongodb.com/docs/manual/reference/command/aggregate/) command using a write stage (e.g. `$out`, +`$merge`). Drivers MUST NOT add a transaction ID to these commands and MUST NOT retry these commands if they fail to +return a response. + +#### Retryable Writes Within Transactions + +In MongoDB 4.0 the only supported retryable write commands within a transaction are `commitTransaction` and +`abortTransaction`. Therefore drivers MUST NOT retry write commands within transactions even when `retryWrites` has been +set to true on the `MongoClient`. In addition, drivers MUST NOT add the `RetryableWriteError` label to any error that +occurs during a write command within a transaction (excepting `commitTransation` and `abortTransaction`), even when +`retryWrites` has been set to true on the `MongoClient`. + +### Implementing Retryable Writes + +#### Determining Retryable Errors + +When connected to a MongoDB instance that supports retryable writes (versions 3.6+), the driver MUST treat all errors +with the RetryableWriteError label as retryable. This error label can be found in the top-level "errorLabels" field of +the error. + +##### RetryableWriteError Labels The RetryableWriteError label might be added to an error in a variety of ways: -- When the driver encounters a network error establishing an initial connection to a server, - it MUST add a RetryableWriteError label to that error if the MongoClient performing - the operation has the retryWrites configuration option set to true. +- When the driver encounters a network error establishing an initial connection to a server, it MUST add a + RetryableWriteError label to that error if the MongoClient performing the operation has the retryWrites configuration + option set to true. -- When the driver encounters a network error communicating with any server - version that supports retryable writes, it MUST add a RetryableWriteError - label to that error if the MongoClient performing the operation has the - retryWrites configuration option set to true. +- When the driver encounters a network error communicating with any server version that supports retryable writes, it + MUST add a RetryableWriteError label to that error if the MongoClient performing the operation has the retryWrites + configuration option set to true. -- When a CMAP-compliant driver encounters a `PoolClearedError`_ during - connection check out, it MUST add a RetryableWriteError label to that error if - the MongoClient performing the operation has the retryWrites configuration +- When a CMAP-compliant driver encounters a [PoolClearedError](<>) during connection check out, it MUST add a + RetryableWriteError label to that error if the MongoClient performing the operation has the retryWrites configuration option set to true. - .. _PoolClearedError: ../connection-monitoring-and-pooling/connection-monitoring-and-pooling.md#connection-pool-errors - -- For server versions 4.4 and newer, the server will add a RetryableWriteError - label to errors or server responses that it considers retryable before - returning them to the driver. As new server versions are released, the errors - that are labeled with the RetryableWriteError label may change. Drivers MUST - NOT add a RetryableWriteError label to any error derived from a 4.4+ server - response (i.e. any error that is not a network error). - -- When receiving a command result with an error from a pre-4.4 server that - supports retryable writes, the driver MUST add a RetryableWriteError label to - errors that meet the following criteria if the retryWrites option is set to - true on the client performing the relevant operation: - - - a mongod or mongos response with any the following error codes in the - top-level ``code`` field: - - .. list-table:: - :header-rows: 1 - - * - Error Name - - Error Code - * - InterruptedAtShutdown - - 11600 - * - InterruptedDueToReplStateChange - - 11602 - * - NotWritablePrimary - - 10107 - * - NotPrimaryNoSecondaryOk - - 13435 - * - NotPrimaryOrSecondary - - 13436 - * - PrimarySteppedDown - - 189 - * - ShutdownInProgress - - 91 - * - HostNotFound - - 7 - * - HostUnreachable - - 6 - * - NetworkTimeout - - 89 - * - SocketException - - 9001 - * - ExceededTimeLimit - - 262 - - - a mongod response with any of the previously listed codes in the - ``writeConcernError.code`` field. +- For server versions 4.4 and newer, the server will add a RetryableWriteError label to errors or server responses that + it considers retryable before returning them to the driver. As new server versions are released, the errors that are + labeled with the RetryableWriteError label may change. Drivers MUST NOT add a RetryableWriteError label to any error + derived from a 4.4+ server response (i.e. any error that is not a network error). + +- When receiving a command result with an error from a pre-4.4 server that supports retryable writes, the driver MUST + add a RetryableWriteError label to errors that meet the following criteria if the retryWrites option is set to true on + the client performing the relevant operation: + + - a mongod or mongos response with any the following error codes in the top-level `code` field: + + | Error Name | Error Code | + | ------------------------------- | ---------- | + | InterruptedAtShutdown | 11600 | + | InterruptedDueToReplStateChange | 11602 | + | NotWritablePrimary | 10107 | + | NotPrimaryNoSecondaryOk | 13435 | + | NotPrimaryOrSecondary | 13436 | + | PrimarySteppedDown | 189 | + | ShutdownInProgress | 91 | + | HostNotFound | 7 | + | HostUnreachable | 6 | + | NetworkTimeout | 89 | + | SocketException | 9001 | + | ExceededTimeLimit | 262 | + + - a mongod response with any of the previously listed codes in the `writeConcernError.code` field. Drivers MUST NOT add a RetryableWriteError label based on the following: - - any ``writeErrors[].code`` fields in a mongod or mongos response - - - the ``writeConcernError.code`` field in a mongos response + - any `writeErrors[].code` fields in a mongod or mongos response + - the `writeConcernError.code` field in a mongos response - The criteria for retryable errors is similar to the discussion in the SDAM - spec's section on `Error Handling`_, but includes additional error codes. See - `What do the additional error codes mean?`_ for the reasoning behind these - additional errors. + The criteria for retryable errors is similar to the discussion in the SDAM spec's section on + [Error Handling](../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling), but includes + additional error codes. See [What do the additional error codes mean?](#what-do-the-additional-error-codes-mean) for + the reasoning behind these additional errors. -To understand why the driver should only add the RetryableWriteError label to an -error when the retryWrites option is true on the MongoClient performing the -operation, see `Why does the driver only add the RetryableWriteError label to -errors that occur on a MongoClient with retryWrites set to true?`_ +To understand why the driver should only add the RetryableWriteError label to an error when the retryWrites option is +true on the MongoClient performing the operation, see +[Why does the driver only add the RetryableWriteError label to errors that occur on a MongoClient with retryWrites set to true?](#why-does-the-driver-only-add-the-retryablewriteerror-label-to-errors-that-occur-on-a-mongoclient-with-retrywrites-set-to-true) -Note: During a retryable write operation on a sharded cluster, mongos may retry -the operation internally, in which case it will not add a RetryableWriteError -label to any error that occurs after those internal retries to prevent excessive +Note: During a retryable write operation on a sharded cluster, mongos may retry the operation internally, in which case +it will not add a RetryableWriteError label to any error that occurs after those internal retries to prevent excessive retrying. -For more information about error labels, see the `Transactions specification`_. +For more information about error labels, see the +[Transactions specification](../transactions/transactions.md#error-labels). + +#### Generating Transaction IDs + +The server requires each retryable write operation to provide a unique transaction ID in its command document. The +transaction ID consists of a server session ID and a monotonically increasing transaction number. The session ID is +obtained from the ClientSession object, which will have either been passed to the write operation from the application +or constructed internally for the operation. Drivers will be responsible for maintaining a monotonically increasing +transaction number for each server session used by a ClientSession object. Drivers that pool server sessions MUST +preserve the transaction number when reusing a server session from the pool with a new ClientSession (this can be +tracked as another property on the driver's object for the server session). -.. _Error Handling: ../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling -.. _Transactions specification: ../transactions/transactions.md#error-labels +Drivers MUST ensure that each retryable write command specifies a transaction number larger than any previously used +transaction number for its session ID. -Generating Transaction IDs -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Since ClientSession objects are not thread safe and may only be used by one thread at a time, drivers should not need to +worry about race conditions when incrementing the transaction number. -The server requires each retryable write operation to provide a unique -transaction ID in its command document. The transaction ID consists of a server -session ID and a monotonically increasing transaction number. The session ID is -obtained from the ClientSession object, which will have either been passed to -the write operation from the application or constructed internally for the -operation. Drivers will be responsible for maintaining a monotonically -increasing transaction number for each server session used by a ClientSession -object. Drivers that pool server sessions MUST preserve the transaction number -when reusing a server session from the pool with a new ClientSession (this can -be tracked as another property on the driver's object for the server session). +#### Behavioral Changes for Write Commands -Drivers MUST ensure that each retryable write command specifies a transaction -number larger than any previously used transaction number for its session ID. +Drivers MUST automatically add a transaction ID to all supported write commands executed via a specific +[CRUD](../crud/crud.md) method (e.g. `updateOne()`) or write command method (e.g. `executeWriteCommand()`) within a +MongoClient where retryable writes have been enabled and when the selected server supports retryable writes. -Since ClientSession objects are not thread safe and may only be used by one -thread at a time, drivers should not need to worry about race conditions when -incrementing the transaction number. +If your driver offers a generic command method on your database object (e.g. `runCommand()`), it MUST NOT check the +user's command document to determine if it is a supported write operation and MUST NOT automatically add a transaction +ID. The method should send the user's command document to the server as-is. -Behavioral Changes for Write Commands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This specification does not affect write commands executed within a MongoClient where retryable writes have not been +enabled. -Drivers MUST automatically add a transaction ID to all supported write commands -executed via a specific `CRUD`_ method (e.g. ``updateOne()``) or write command -method (e.g. ``executeWriteCommand()``) within a MongoClient where retryable -writes have been enabled and when the selected server supports retryable writes. +#### Constructing Write Commands -.. _CRUD: ../crud/crud.md +When constructing a supported write command that will be executed within a MongoClient where retryable writes have been +enabled, drivers MUST increment the transaction number for the corresponding server session and include the server +session ID and transaction number in top-level `lsid` and `txnNumber` fields, respectively. `lsid` is a BSON value +(discussed in the [Driver Session](../sessions/driver-sessions.rst) specification). `txnNumber` MUST be a positive +64-bit integer (BSON type 0x12). -If your driver offers a generic command method on your database object (e.g. -``runCommand()``), it MUST NOT check the user's command document to determine if -it is a supported write operation and MUST NOT automatically add a transaction -ID. The method should send the user's command document to the server as-is. +The following example illustrates a possible write command for an `updateOne()` operation: -This specification does not affect write commands executed within a MongoClient -where retryable writes have not been enabled. - -Constructing Write Commands -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When constructing a supported write command that will be executed within a -MongoClient where retryable writes have been enabled, drivers MUST increment the -transaction number for the corresponding server session and include the server -session ID and transaction number in top-level ``lsid`` and ``txnNumber`` -fields, respectively. ``lsid`` is a BSON value (discussed in the -`Driver Session`_ specification). ``txnNumber`` MUST be a positive 64-bit -integer (BSON type 0x12). - -The following example illustrates a possible write command for an -``updateOne()`` operation: - -.. code:: typescript - - { - update: "coll", - lsid: { ... }, - txnNumber: 100, - updates: [ - { q: { x: 1 }, u: { $inc: { y: 1 } } }, - ], - ordered: true - } +```typescript +{ + update: "coll", + lsid: { ... }, + txnNumber: 100, + updates: [ + { q: { x: 1 }, u: { $inc: { y: 1 } } }, + ], + ordered: true +} +``` -When constructing multiple write commands for a multi-statement write operation -(i.e. ``insertMany()`` and ``bulkWrite()``), drivers MUST increment the -transaction number for each supported write command in the batch. +When constructing multiple write commands for a multi-statement write operation (i.e. `insertMany()` and `bulkWrite()`), +drivers MUST increment the transaction number for each supported write command in the batch. -Executing Retryable Write Commands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#### Executing Retryable Write Commands -When selecting a writable server for the first attempt of a retryable write -command, drivers MUST allow a server selection error to propagate. In this case, -the caller is able to infer that no attempt was made. +When selecting a writable server for the first attempt of a retryable write command, drivers MUST allow a server +selection error to propagate. In this case, the caller is able to infer that no attempt was made. -If retryable writes is not enabled or the selected server does not support -retryable writes, drivers MUST NOT include a transaction ID in the command and -MUST attempt to execute the write command exactly once and allow any errors to +If retryable writes is not enabled or the selected server does not support retryable writes, drivers MUST NOT include a +transaction ID in the command and MUST attempt to execute the write command exactly once and allow any errors to propagate. In this case, the caller is able to infer that an attempt was made. -If retryable writes are enabled and the selected server supports retryable -writes, drivers MUST add a transaction ID to the command. Drivers MUST only -attempt to retry a write command if the first attempt yields a retryable error. -Drivers MUST NOT attempt to retry a write command on any other error. +If retryable writes are enabled and the selected server supports retryable writes, drivers MUST add a transaction ID to +the command. Drivers MUST only attempt to retry a write command if the first attempt yields a retryable error. Drivers +MUST NOT attempt to retry a write command on any other error. -If the first attempt of a write command including a transaction ID encounters -a retryable error, the driver MUST update its topology according to the SDAM -spec (see: `Error Handling`_) and capture this original retryable error. +If the first attempt of a write command including a transaction ID encounters a retryable error, the driver MUST update +its topology according to the SDAM spec (see: +[Error Handling](../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling)) and capture +this original retryable error. -Drivers MUST then retry the operation as many times as necessary until any one -of the following conditions is reached: +Drivers MUST then retry the operation as many times as necessary until any one of the following conditions is reached: - the operation succeeds. - the operation fails with a non-retryable error. -- CSOT is enabled and the operation times out per `Client Side - Operations Timeout: Retryability - <../client-side-operations-timeout/client-side-operations-timeout.md#retryability>`__. +- CSOT is enabled and the operation times out per + [Client Side Operations Timeout: Retryability](../client-side-operations-timeout/client-side-operations-timeout.md#retryability). - CSOT is not enabled and one retry was attempted. -For each retry attempt, drivers MUST select a writable server. In a sharded -cluster, the server on which the operation failed MUST be provided to -the server selection mechanism as a deprioritized server. +For each retry attempt, drivers MUST select a writable server. In a sharded cluster, the server on which the operation +failed MUST be provided to the server selection mechanism as a deprioritized server. -If the driver cannot select a server for a retry attempt -or the selected server does not support retryable writes, retrying is not -possible and drivers MUST raise the retryable error from the previous attempt. -In both cases, the caller is able to infer that an attempt was made. +If the driver cannot select a server for a retry attempt or the selected server does not support retryable writes, +retrying is not possible and drivers MUST raise the retryable error from the previous attempt. In both cases, the caller +is able to infer that an attempt was made. -If a retry attempt also fails, drivers MUST update their topology according to -the SDAM spec (see: `Error Handling`_). If an error would not allow the caller -to infer that an attempt was made (e.g. connection pool exception originating -from the driver) or the error is labeled "NoWritesPerformed", the error from -the previous attempt should be raised. If all server errors are labeled -"NoWritesPerformed", then the first error should be raised. +If a retry attempt also fails, drivers MUST update their topology according to the SDAM spec (see: +[Error Handling](../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling)). If an error +would not allow the caller to infer that an attempt was made (e.g. connection pool exception originating from the +driver) or the error is labeled "NoWritesPerformed", the error from the previous attempt should be raised. If all server +errors are labeled "NoWritesPerformed", then the first error should be raised. -If a driver associates server information (e.g. the server address or -description) with an error, the driver MUST ensure that the reported server -information corresponds to the server that originated the error. +If a driver associates server information (e.g. the server address or description) with an error, the driver MUST ensure +that the reported server information corresponds to the server that originated the error. The above rules are implemented in the following pseudo-code: -.. code-block:: typescript - - /** - * Checks if a server supports retryable writes. - */ - function isRetryableWritesSupported(server) { - if (server.getMaxWireVersion() < RETRYABLE_WIRE_VERSION) { - return false; - } - - if ( ! server.hasLogicalSessionTimeoutMinutes()) { - return false; - } - - if (server.isStandalone()) { - return false; - } - - return true; +```typescript +/** + * Checks if a server supports retryable writes. + */ +function isRetryableWritesSupported(server) { + if (server.getMaxWireVersion() < RETRYABLE_WIRE_VERSION) { + return false; } - /** - * Executes a write command in the context of a MongoClient where retryable - * writes have been enabled. The session parameter may be an implicit or - * explicit client session (depending on how the CRUD method was invoked). - */ - function executeRetryableWrite(command, session) { - /* Allow ServerSelectionException to propagate to our caller, which can then - * assume that no attempts were made. */ - server = selectServer("writable"); - - /* If the server does not support retryable writes, execute the write as if - * retryable writes are not enabled. */ - if ( ! isRetryableWritesSupported(server)) { - return executeCommand(server, command); - } + if ( ! server.hasLogicalSessionTimeoutMinutes()) { + return false; + } - /* Incorporate lsid and txnNumber fields into the command document. These - * values will be derived from the implicit or explicit session object. */ - retryableCommand = addTransactionIdToCommand(command, session); - - Exception previousError = null; - retrying = false; - while true { - try { - return executeCommand(server, retryableCommand); - } catch (Exception currentError) { - handleError(currentError); - - /* If the error has a RetryableWriteError label, remember the exception - * and proceed with retrying the operation. - * - * IllegalOperation (code 20) with errmsg starting with "Transaction - * numbers" MUST be re-raised with an actionable error message. - */ - if (!currentError.hasErrorLabel("RetryableWriteError")) { - if ( currentError.code == 20 && previousError.errmsg.startsWith("Transaction numbers") ) { - currentError.errmsg = "This MongoDB deployment does not support retryable..."; - } - throw currentError; - } + if (server.isStandalone()) { + return false; + } - /* - * If the "previousError" is "null", then the "currentError" is the - * first error encountered during the retry attempt cycle. We must - * persist the first error in the case where all succeeding errors are - * labeled "NoWritesPerformed", which would otherwise raise "null" as - * the error. - */ - if (previousError == null) { - previousError = currentError; - } + return true; +} + +/** + * Executes a write command in the context of a MongoClient where retryable + * writes have been enabled. The session parameter may be an implicit or + * explicit client session (depending on how the CRUD method was invoked). + */ +function executeRetryableWrite(command, session) { + /* Allow ServerSelectionException to propagate to our caller, which can then + * assume that no attempts were made. */ + server = selectServer("writable"); + + /* If the server does not support retryable writes, execute the write as if + * retryable writes are not enabled. */ + if ( ! isRetryableWritesSupported(server)) { + return executeCommand(server, command); + } - /* - * For exceptions that originate from the driver (e.g. no socket available - * from the connection pool), we should raise the previous error if there - * was one. - */ - if (currentError is not DriverException && ! previousError.hasErrorLabel("NoWritesPerformed")) { - previousError = currentError; + /* Incorporate lsid and txnNumber fields into the command document. These + * values will be derived from the implicit or explicit session object. */ + retryableCommand = addTransactionIdToCommand(command, session); + + Exception previousError = null; + retrying = false; + while true { + try { + return executeCommand(server, retryableCommand); + } catch (Exception currentError) { + handleError(currentError); + + /* If the error has a RetryableWriteError label, remember the exception + * and proceed with retrying the operation. + * + * IllegalOperation (code 20) with errmsg starting with "Transaction + * numbers" MUST be re-raised with an actionable error message. + */ + if (!currentError.hasErrorLabel("RetryableWriteError")) { + if ( currentError.code == 20 && previousError.errmsg.startsWith("Transaction numbers") ) { + currentError.errmsg = "This MongoDB deployment does not support retryable..."; } + throw currentError; } /* - * We try to select server that is not the one that failed by passing the - * failed server as a deprioritized server. - * If we cannot select a writable server, do not proceed with retrying and - * throw the previous error. The caller can then infer that an attempt was - * made and failed. */ - try { - deprioritizedServers = [ server ]; - server = selectServer("writable", deprioritizedServers); - } catch (Exception ignoredError) { - throw previousError; + * If the "previousError" is "null", then the "currentError" is the + * first error encountered during the retry attempt cycle. We must + * persist the first error in the case where all succeeding errors are + * labeled "NoWritesPerformed", which would otherwise raise "null" as + * the error. + */ + if (previousError == null) { + previousError = currentError; } - /* If the server selected for retrying is too old, throw the previous error. - * The caller can then infer that an attempt was made and failed. This case - * is very rare, and likely means that the cluster is in the midst of a - * downgrade. */ - if ( ! isRetryableWritesSupported(server)) { - throw previousError; + /* + * For exceptions that originate from the driver (e.g. no socket available + * from the connection pool), we should raise the previous error if there + * was one. + */ + if (currentError is not DriverException && ! previousError.hasErrorLabel("NoWritesPerformed")) { + previousError = currentError; } + } - if (timeoutMS == null) { - /* If CSOT is not enabled, allow any retryable error from the second - * attempt to propagate to our caller, as it will be just as relevant - * (if not more relevant) than the original error. */ - if (retrying) { - throw previousError; - } - } else if (isExpired(timeoutMS)) { - /* CSOT is enabled and the operation has timed out. */ + /* + * We try to select server that is not the one that failed by passing the + * failed server as a deprioritized server. + * If we cannot select a writable server, do not proceed with retrying and + * throw the previous error. The caller can then infer that an attempt was + * made and failed. */ + try { + deprioritizedServers = [ server ]; + server = selectServer("writable", deprioritizedServers); + } catch (Exception ignoredError) { + throw previousError; + } + + /* If the server selected for retrying is too old, throw the previous error. + * The caller can then infer that an attempt was made and failed. This case + * is very rare, and likely means that the cluster is in the midst of a + * downgrade. */ + if ( ! isRetryableWritesSupported(server)) { + throw previousError; + } + + if (timeoutMS == null) { + /* If CSOT is not enabled, allow any retryable error from the second + * attempt to propagate to our caller, as it will be just as relevant + * (if not more relevant) than the original error. */ + if (retrying) { throw previousError; } - retrying = true; + } else if (isExpired(timeoutMS)) { + /* CSOT is enabled and the operation has timed out. */ + throw previousError; } + retrying = true; } +} +``` + +`handleError` in the above pseudocode refers to the function defined in the +[Error handling pseudocode](../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling-pseudocode) +section of the SDAM specification. + +When retrying a write command, drivers MUST resend the command with the same transaction ID. Drivers MUST NOT resend the +original wire protocol message if doing so would violate rules for +[gossipping the cluster time](../sessions/driver-sessions.rst#gossipping-the-cluster-time) (see: +[Can drivers resend the same wire protocol message on retry attempts?](#can-drivers-resend-the-same-wire-protocol-message-on-retry-attempts)). + +In the case of a multi-statement write operation split across multiple write commands, a failed retry attempt will also +interrupt execution of any additional write operations in the batch (regardless of the ordered option). This is no +different than if a retryable error had been encountered without retryable behavior enabled or supported by the driver. +Drivers are encouraged to provide access to an intermediary write result (e.g. BulkWriteResult, InsertManyResult) +through the BulkWriteException, in accordance with the [CRUD](../crud/crud.md) specification. + +## Logging Retry Attempts + +Drivers MAY choose to log retry attempts for write operations. This specification does not define a format for such log +messages. + +## Command Monitoring + +In accordance with the +[Command Logging and Monitoring](../command-logging-and-monitoring/command-logging-and-monitoring.rst) specification, +drivers MUST guarantee that each `CommandStartedEvent` has either a correlating `CommandSucceededEvent` or +`CommandFailedEvent` and that every "command started" log message has either a correlating "command succeeded" log +message or "command failed" log message. If the first attempt of a retryable write operation encounters a retryable +error, drivers MUST fire a `CommandFailedEvent` and emit a "command failed" log message for the retryable error and fire +a separate `CommandStartedEvent` and "command succeeded" log message when executing the subsequent retry attempt. Note +that the second `CommandStartedEvent` and "command succeeded" log message may have a different `connectionId`, since a +writable server is reselected for the retry attempt. + +Each attempt of a retryable write operation SHOULD report a different `requestId` so that events for each attempt can be +properly correlated with one another. -``handleError`` in the above pseudocode refers to the function defined in the -`Error handling pseudocode`_ section of the SDAM specification. - -.. _Error handling pseudocode: ../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#error-handling-pseudocode - -When retrying a write command, drivers MUST resend the command with the same -transaction ID. Drivers MUST NOT resend the original wire protocol message if -doing so would violate rules for `gossipping the cluster time`_ (see: -`Can drivers resend the same wire protocol message on retry attempts?`_). - -.. _gossipping the cluster time: ../sessions/driver-sessions.rst#gossipping-the-cluster-time - -In the case of a multi-statement write operation split across multiple write -commands, a failed retry attempt will also interrupt execution of any additional -write operations in the batch (regardless of the ordered option). This is no -different than if a retryable error had been encountered without retryable -behavior enabled or supported by the driver. Drivers are encouraged to provide -access to an intermediary write result (e.g. BulkWriteResult, InsertManyResult) -through the BulkWriteException, in accordance with the `CRUD`_ specification. - -Logging Retry Attempts -====================== - -Drivers MAY choose to log retry attempts for write operations. This -specification does not define a format for such log messages. - -Command Monitoring -================== - -In accordance with the `Command Logging and Monitoring`_ specification, drivers MUST -guarantee that each ``CommandStartedEvent`` has either a correlating -``CommandSucceededEvent`` or ``CommandFailedEvent`` and that every "command started" -log message has either a correlating "command succeeded" log message or "command failed" -log message. If the first attempt of a retryable write operation encounters a retryable -error, drivers MUST fire a ``CommandFailedEvent`` and emit a "command failed" log message for the retryable error and fire a -separate ``CommandStartedEvent`` and "command succeeded" log message when executing the subsequent retry attempt. Note that -the second ``CommandStartedEvent`` and "command succeeded" log message may have a different ``connectionId``, since -a writable server is reselected for the retry attempt. - -.. _Command Logging and Monitoring: ../command-logging-and-monitoring/command-logging-and-monitoring.rst - -Each attempt of a retryable write operation SHOULD report a different -``requestId`` so that events for each attempt can be properly correlated with -one another. - -The `Command Logging and Monitoring`_ specification states that the ``operationId`` field is -a driver-generated, 64-bit integer and may be "used to link events together such -as bulk write operations." Each attempt of a retryable write operation SHOULD -report the same ``operationId``; however, drivers SHOULD NOT use the -``operationId`` field to relay information about a transaction ID. A bulk write -operation may consist of multiple write commands, each of which may specify a -unique transaction ID. - -Test Plan -========= - -See the `README `_ for tests. - -At a high level, the test plan will cover the following scenarios for executing -supported write operations within a MongoClient where retryable writes have been -enabled: - -* Executing the same write operation (and transaction ID) multiple times should - yield an identical write result. -* Test at-most-once behavior by observing that subsequent executions of the same - write operation do not incur further modifications to the collection data. -* Exercise supported single-statement write operations (i.e. deleteOne, - insertOne, replaceOne, updateOne, and findAndModify). -* Exercise supported multi-statement insertMany and bulkWrite operations, which - contain only supported single-statement write operations. Both ordered and - unordered execution should be tested. +The [Command Logging and Monitoring](../command-logging-and-monitoring/command-logging-and-monitoring.rst) specification +states that the `operationId` field is a driver-generated, 64-bit integer and may be "used to link events together such +as bulk write operations." Each attempt of a retryable write operation SHOULD report the same `operationId`; however, +drivers SHOULD NOT use the `operationId` field to relay information about a transaction ID. A bulk write operation may +consist of multiple write commands, each of which may specify a unique transaction ID. + +## Test Plan + +See the [README](tests/README.md) for tests. + +At a high level, the test plan will cover the following scenarios for executing supported write operations within a +MongoClient where retryable writes have been enabled: + +- Executing the same write operation (and transaction ID) multiple times should yield an identical write result. +- Test at-most-once behavior by observing that subsequent executions of the same write operation do not incur further + modifications to the collection data. +- Exercise supported single-statement write operations (i.e. deleteOne, insertOne, replaceOne, updateOne, and + findAndModify). +- Exercise supported multi-statement insertMany and bulkWrite operations, which contain only supported single-statement + write operations. Both ordered and unordered execution should be tested. Additional prose tests for other scenarios are also included. -Motivation for Change -===================== +## Motivation for Change -Drivers currently have no API for specifying at-most-once semantics and -retryable behavior for write operations. The driver API needs to be extended to -support this behavior. +Drivers currently have no API for specifying at-most-once semantics and retryable behavior for write operations. The +driver API needs to be extended to support this behavior. -Design Rationale -================ +## Design Rationale -The design of this specification piggy-backs that of the `Driver Session`_ -specification in that it modifies the driver API as little as possible to -introduce the concept of at-most-once semantics and retryable behavior for write -operations. A transaction ID will be included in all supported write commands -executed within the scope of a MongoClient where retryable writes have been -enabled. +The design of this specification piggy-backs that of the [Driver Session](../sessions/driver-sessions.rst) specification +in that it modifies the driver API as little as possible to introduce the concept of at-most-once semantics and +retryable behavior for write operations. A transaction ID will be included in all supported write commands executed +within the scope of a MongoClient where retryable writes have been enabled. + +Drivers expect the server to yield an error if a transaction ID is included in an unsupported write command. This +requires drivers to maintain an allow list and track which write operations support retryable behavior for a given +server version (see: +[Why must drivers maintain an allow list of supported operations?](#why-must-drivers-maintain-an-allow-list-of-supported-operations)). + +While this approach will allow applications to take advantage of retryable write behavior with minimal code changes, it +also presents a documentation challenge. Users must understand exactly what can and will be retried (see: +[How will users know which operations are supported?](#how-will-users-know-which-operations-are-supported)). + +## Backwards Compatibility + +The API changes to support retryable writes extend the existing API but do not introduce any backward breaking changes. +Existing programs that do not make use of retryable writes will continue to compile and run correctly. -Drivers expect the server to yield an error if a transaction ID is included in -an unsupported write command. This requires drivers to maintain an allow list and -track which write operations support retryable behavior for a given server -version (see: `Why must drivers maintain an allow list of supported -operations?`_). - -While this approach will allow applications to take advantage of retryable write -behavior with minimal code changes, it also presents a documentation challenge. -Users must understand exactly what can and will be retried (see: `How will users -know which operations are supported?`_). - -Backwards Compatibility -======================= - -The API changes to support retryable writes extend the existing API but do not -introduce any backward breaking changes. Existing programs that do not make use -of retryable writes will continue to compile and run correctly. - -Reference Implementation -======================== - -The C# and C drivers will provide reference implementations. JIRA links will be -added here at a later point. - -Future Work -=========== - -Supporting at-most-once semantics and retryable behavior for updateMany and -deleteMany operations may become possible once the server implements support for -multi-document transactions. - -A separate specification for retryable read operations could complement this -specification. Retrying read operations would not require client or server -sessions and could be implemented independently of retryable writes. - -Q & A -===== - -What do the additional error codes mean? ----------------------------------------- - -The errors `HostNotFound`, `HostUnreachable`, `NetworkTimeout`, -`SocketException` may be returned from mongos during problems routing to a -shard. These may be transient, or localized to that mongos. - -Why are write operations only retried once by default? ------------------------------------------------------- - -The spec concerns itself with retrying write operations that encounter a -retryable error (i.e. no response due to network error or a response indicating -that the node is no longer a primary). A retryable error may be classified as -either a transient error (e.g. dropped connection, replica set failover) or -persistent outage. In the case of a transient error, the driver will mark the -server as "unknown" per the `SDAM`_ spec. A subsequent retry attempt will allow -the driver to rediscover the primary within the designated server selection -timeout period (30 seconds by default). If server selection times out during -this retry attempt, we can reasonably assume that there is a persistent outage. -In the case of a persistent outage, multiple retry attempts are fruitless and -would waste time. See `How To Write Resilient MongoDB Applications`_ for -additional discussion on this strategy. - -However when `Client Side Operations Timeout`_ is enabled, the driver will -retry multiple times until the operation succeeds, a non-retryable error -is encountered, or the timeout expires. Retrying multiple times provides -greater resilience to cascading failures such as rolling server restarts -during planned maintenance events. - -.. _SDAM: ../server-discovery-and-monitoring/server-discovery-and-monitoring.rst -.. _How To Write Resilient MongoDB Applications: https://emptysqua.re/blog/how-to-write-resilient-mongodb-applications/ -.. _Client Side Operations Timeout: ../client-side-operations-timeout/client-side-operations-timeout.rst - -What if the transaction number overflows? ------------------------------------------ - -Since server sessions are pooled and session lifetimes are configurable on -the server, it is theoretically possible for the transaction number to overflow -if it reaches the limits of a signed 64-bit integer. The spec does not address -this scenario. Drivers may decide to handle this as they wish. For example, they -may raise a client-side error if a transaction number would overflow, eagerly -remove sessions with sufficiently high transactions numbers from the pool in an -attempt to limit such occurrences, or simply rely on the server to raise an -error when a transaction number is reused. - -Why are unacknowledged write concerns unsupported? --------------------------------------------------- - -The server does not consider the write concern when deciding if a write -operation supports retryable behavior. Technically, operations with an -unacknowledged write concern can specify a transaction ID and be retried. -However, the spec elects not to support unacknowledged write concerns due to -various ways that drivers may issue write operations with unacknowledged write -concerns. - -When using ``OP_QUERY`` to issue a write command to the server, a command -response is always returned. A write command with an unacknowledged write -concern (i.e. ``{w: 0}``) will return a response of ``{ok: 1}``. If a retryable -error is encountered (either a network error or "not writeable primary" response), -the driver could attempt to retry the operation by executing it again with the same -transaction ID. - -Some drivers fall back to legacy opcodes (e.g. ``OP_INSERT``) to execute write -operations with an unacknowledged write concern. In the future, ``OP_MSG`` may -allow the server to avoid returning any response for write operations sent with -an unacknowledged write concern. In both of these cases, there is no response -for which the driver might encounter a retryable error and decide to retry the -operation. - -Rather than depend on an implementation detail to determine if retryable -behavior might apply, the spec has chosen to not support retryable behavior -for unacknowledged write concerns and guarantee a consistent user experience -across all drivers. - -Why must drivers maintain an allow list of supported operations? ----------------------------------------------------------------- - -Requiring that drivers maintain an allow list of supported write operations is -unfortunate. It both adds complexity to the driver's implementation and limits -the driver's ability to immediately take advantage of new server functionality +## Reference Implementation + +The C# and C drivers will provide reference implementations. JIRA links will be added here at a later point. + +## Future Work + +Supporting at-most-once semantics and retryable behavior for updateMany and deleteMany operations may become possible +once the server implements support for multi-document transactions. + +A separate specification for retryable read operations could complement this specification. Retrying read operations +would not require client or server sessions and could be implemented independently of retryable writes. + +## Q & A + +### What do the additional error codes mean? + +The errors `HostNotFound`, `HostUnreachable`, `NetworkTimeout`, `SocketException` may be returned from mongos during +problems routing to a shard. These may be transient, or localized to that mongos. + +### Why are write operations only retried once by default? + +The spec concerns itself with retrying write operations that encounter a retryable error (i.e. no response due to +network error or a response indicating that the node is no longer a primary). A retryable error may be classified as +either a transient error (e.g. dropped connection, replica set failover) or persistent outage. In the case of a +transient error, the driver will mark the server as "unknown" per the +[SDAM](../server-discovery-and-monitoring/server-discovery-and-monitoring.rst) spec. A subsequent retry attempt will +allow the driver to rediscover the primary within the designated server selection timeout period (30 seconds by +default). If server selection times out during this retry attempt, we can reasonably assume that there is a persistent +outage. In the case of a persistent outage, multiple retry attempts are fruitless and would waste time. See +[How To Write Resilient MongoDB Applications](https://emptysqua.re/blog/how-to-write-resilient-mongodb-applications/) +for additional discussion on this strategy. + +However when [Client Side Operations Timeout](../client-side-operations-timeout/client-side-operations-timeout.rst) is +enabled, the driver will retry multiple times until the operation succeeds, a non-retryable error is encountered, or the +timeout expires. Retrying multiple times provides greater resilience to cascading failures such as rolling server +restarts during planned maintenance events. + +### What if the transaction number overflows? + +Since server sessions are pooled and session lifetimes are configurable on the server, it is theoretically possible for +the transaction number to overflow if it reaches the limits of a signed 64-bit integer. The spec does not address this +scenario. Drivers may decide to handle this as they wish. For example, they may raise a client-side error if a +transaction number would overflow, eagerly remove sessions with sufficiently high transactions numbers from the pool in +an attempt to limit such occurrences, or simply rely on the server to raise an error when a transaction number is +reused. + +### Why are unacknowledged write concerns unsupported? + +The server does not consider the write concern when deciding if a write operation supports retryable behavior. +Technically, operations with an unacknowledged write concern can specify a transaction ID and be retried. However, the +spec elects not to support unacknowledged write concerns due to various ways that drivers may issue write operations +with unacknowledged write concerns. + +When using `OP_QUERY` to issue a write command to the server, a command response is always returned. A write command +with an unacknowledged write concern (i.e. `{w: 0}`) will return a response of `{ok: 1}`. If a retryable error is +encountered (either a network error or "not writeable primary" response), the driver could attempt to retry the +operation by executing it again with the same transaction ID. + +Some drivers fall back to legacy opcodes (e.g. `OP_INSERT`) to execute write operations with an unacknowledged write +concern. In the future, `OP_MSG` may allow the server to avoid returning any response for write operations sent with an +unacknowledged write concern. In both of these cases, there is no response for which the driver might encounter a +retryable error and decide to retry the operation. + +Rather than depend on an implementation detail to determine if retryable behavior might apply, the spec has chosen to +not support retryable behavior for unacknowledged write concerns and guarantee a consistent user experience across all +drivers. + +### Why must drivers maintain an allow list of supported operations? + +Requiring that drivers maintain an allow list of supported write operations is unfortunate. It both adds complexity to +the driver's implementation and limits the driver's ability to immediately take advantage of new server functionality (i.e. the driver must be upgraded to support additional write operations). Several other alternatives were discussed: -* The server could inform drivers which write operations support retryable - behavior in its ``hello`` or legacy hello response. This would be a form of - feature discovery, for which there is no established protocol. It would also - add complexity to the connection handshake. -* The server could ignore a transaction ID on the first observed attempt of an - unsupported write command and only yield an error on subsequent attempts. This - would require the server to create a transaction record for unsupported writes - to avoid the risk of applying a write twice and ensuring that retry attempts - could be differentiated. It also poses a significant problem for sharding if a - multi-document write does not reach all shards, since those shards would not +- The server could inform drivers which write operations support retryable behavior in its `hello` or legacy hello + response. This would be a form of feature discovery, for which there is no established protocol. It would also add + complexity to the connection handshake. +- The server could ignore a transaction ID on the first observed attempt of an unsupported write command and only yield + an error on subsequent attempts. This would require the server to create a transaction record for unsupported writes + to avoid the risk of applying a write twice and ensuring that retry attempts could be differentiated. It also poses a + significant problem for sharding if a multi-document write does not reach all shards, since those shards would not know to create a transaction record. -* The driver could allow more fine-grained control retryable write behavior by - supporting a ``retryWrites`` option on the database and collection objects. - This would allow users to enable ``retryWrites`` on a MongoClient and disable - it as needed to execute unsupported write operations, or vice versa. Since we - expect the ``retryWrites`` option to become less relevant once transactions - are implemented, we would prefer not to add the option throughout the driver - API. - -How will users know which operations are supported? ---------------------------------------------------- - -The initial list of supported operations is already quite permissive. Most -`CRUD`_ operations are supported apart from ``updateMany()``, ``deleteMany()``, -and ``aggregate()`` with a write stage (e.g. ``$out``, ``$merge``). Other write -operations (e.g. ``renameCollection``) are rare. - -That said, drivers will need to clearly document exactly which operations -support retryable behavior. In the case ``bulkWrite()``, which may or may not -support retryability, drivers should discuss how eligibility is determined. - -Can drivers resend the same wire protocol message on retry attempts? --------------------------------------------------------------------- - -Since retry attempts entail sending the same command and transaction ID to the -server, drivers might consider resending the same wire protocol message in order -to avoid constructing a new message and computing its checksum. The server will -not complain if it receives two messages with the same ``requestId``, as the -field is only used for logging and populating the ``responseTo`` field in its -replies to the client. That said, re-using a wire protocol message might violate -rules for `gossipping the cluster time`_ and might also have implications for -`Command Monitoring`_, since the original write command and its retry attempt -may report the same ``requestId``. - -Why can't drivers split bulk write commands to maximize retryability? ---------------------------------------------------------------------- - -In `Supported Write Operations`_, the spec prohibits drivers from altering -existing logic for splits ``bulkWrite()``'s ``requests`` parameter into write -commands in an attempt to segregate unsupported, multi-document write operations -and maximize retryability for other, supported write operations. The reasoning -behind this prohibition is that such behavior would conflict with a primary goal -of the bulk API in reducing the number of command round-trips to the server. - -retryWrites originally defaulted to false, why does it now default to true? ---------------------------------------------------------------------------- - -Since the initial release of retryable writes in MongoDB 3.6 testing showed -that the overhead for supported operations was sufficiently small that there -was no risk in changing the default. Additionally, the fact that some -operations continue to be unsupported for retryable writes (updateMany and -deleteMany) does not seem to pose a problem in practice. - -Why do drivers have to parse errmsg to determine storage engine support? ------------------------------------------------------------------------- - -There is no reliable way to determine the storage engine in use for shards -in a sharded cluster, and replica sets (and shards) can have mixed deployments -using different storage engines on different members. This is especially -true when a replica set or sharded cluster is being upgraded from one -storage engine to another. This could be common when upgrading to MongoDB -4.2, where MMAPv1 is no longer supported. - -The server returns error code 20 (IllegalOperation) when the storage engine -doesn't support document-level locking and txnNumbers. Error code 20 is used -for a large number of different error cases in the server so we need some other -way to differentiate this error case from any other. The error code and errmsg -are the same in MongoDB 3.6 and 4.0, and the same from a replica set or sharded -cluster (mongos just forwards the error from the shard's replica set). - -Why does the driver only add the RetryableWriteError label to errors that occur on a MongoClient with retryWrites set to true? ------------------------------------------------------------------------------------------------------------------------------- - -The driver does this to maintain consistency with the MongoDB server. -Servers that support the RetryableWriteError label (MongoDB version 4.4 and newer) -only add the label to an error when the client has added a txnNumber to the -command, which only happens when the retryWrites option is true on the client. -For the driver to add the label even if retryWrites is not true would be -inconsistent with the server and potentially confusing to developers. - -Changelog -========= - -:2024-04-29: Fix the link to the Driver Sessions spec. -:2024-01-16: Do not use ``writeConcernError.code`` in pre-4.4 mongos response to - determine retryability. Do not use ``writeErrors[].code`` in - pre-4.4 server responses to determine retryability. -:2023-12-06: Clarify that writes are not retried within transactions. -:2023-12-05: Add that any server information associated with retryable - exceptions MUST reflect the originating server, even in the - presence of retries. -:2023-10-02: When CSOT is not enabled, one retry attempt occurs. -:2023-08-26: Require that in a sharded cluster the server on which the - operation failed MUST be provided to the server selection - mechanism as a deprioritized server. -:2022-11-17: Add logic for persisting "currentError" as "previousError" on first - retry attempt, avoiding raising "null" errors. -:2022-11-09: CLAM must apply both events and log messages. -:2022-10-18: When CSOT is enabled multiple retry attempts may occur. -:2022-10-05: Remove spec front matter and reformat changelog. -:2022-01-25: Note that drivers should retry handshake network failures. -:2021-11-02: Clarify that error labels are only specified in a top-level field - of an error. -:2021-04-26: Replaced deprecated terminology -:2021-03-24: Require that PoolClearedErrors be retried -:2020-09-01: State the the driver should only add the RetryableWriteError label - to network errors when connected to a 4.4+ server. -:2020-02-25: State that the driver should only add the RetryableWriteError label - when retryWrites is on, and make it clear that mongos will - sometimes perform internal retries and not return the - RetryableWriteError label. -:2020-02-10: Remove redundant content in Tests section. -:2020-01-14: Add ExceededTimeLimit to the list of error codes that should - receive a RetryableWriteError label. -:2019-10-21: Change the definition of "retryable write" to be based on the - RetryableWriteError label. Stop requiring drivers to parse errmsg - to categorize retryable errors for pre-4.4 servers. -:2019-07-30: Drivers must rewrite error messages for error code 20 when - txnNumber is not supported by the storage engine. -:2019-06-07: Mention $merge stage for aggregate alongside $out -:2019-05-29: Renamed InterruptedDueToStepDown to InterruptedDueToReplStateChange -:2019-03-06: retryWrites now defaults to true. -:2019-03-05: Prohibit resending wire protocol messages if doing so would violate - rules for gossipping the cluster time. -:2018-06-07: WriteConcernFailed is not a retryable error code. -:2018-04-25: Evaluate retryable eligibility of bulkWrite() commands individually. -:2018-03-14: Clarify that retryable writes may fail with a FCV 3.4 shard. -:2017-11-02: Drivers should not raise errors if selected server does not support - retryable writes and instead fall back to non-retryable behavior. - In addition to wire protocol version, drivers may check for - ``logicalSessionTimeoutMinutes`` to determine if a server supports - sessions and retryable writes. -:2017-10-26: Errors when retrying may be raised instead of the original error - provided they allow the user to infer that an attempt was made. -:2017-10-23: Drivers must document operations that support retryability. -:2017-10-23: Raise the original retryable error if server selection or wire - protocol checks fail during the retry attempt. Encourage drivers to - provide intermediary write results after an unrecoverable failure - during a bulk write. -:2017-10-18: Standalone servers do not support retryable writes. -:2017-10-18: Also retry writes after a "not writable primary" error. -:2017-10-08: Renamed ``txnNum`` to ``txnNumber`` and noted that it must be a - 64-bit integer (BSON type 0x12). -:2017-08-25: Drivers will maintain an allow list so that only supported write - operations may be retried. Transaction IDs will not be included in - unsupported write commands, irrespective of the ``retryWrites`` - option. -:2017-08-18: ``retryWrites`` is now a MongoClient option. +- The driver could allow more fine-grained control retryable write behavior by supporting a `retryWrites` option on the + database and collection objects. This would allow users to enable `retryWrites` on a MongoClient and disable it as + needed to execute unsupported write operations, or vice versa. Since we expect the `retryWrites` option to become less + relevant once transactions are implemented, we would prefer not to add the option throughout the driver API. + +### How will users know which operations are supported? + +The initial list of supported operations is already quite permissive. Most [CRUD](../crud/crud.md) operations are +supported apart from `updateMany()`, `deleteMany()`, and `aggregate()` with a write stage (e.g. `$out`, `$merge`). Other +write operations (e.g. `renameCollection`) are rare. + +That said, drivers will need to clearly document exactly which operations support retryable behavior. In the case +`bulkWrite()`, which may or may not support retryability, drivers should discuss how eligibility is determined. + +### Can drivers resend the same wire protocol message on retry attempts? + +Since retry attempts entail sending the same command and transaction ID to the server, drivers might consider resending +the same wire protocol message in order to avoid constructing a new message and computing its checksum. The server will +not complain if it receives two messages with the same `requestId`, as the field is only used for logging and populating +the `responseTo` field in its replies to the client. That said, re-using a wire protocol message might violate rules for +[gossipping the cluster time](../sessions/driver-sessions.rst#gossipping-the-cluster-time) and might also have +implications for [Command Monitoring](#command-monitoring), since the original write command and its retry attempt may +report the same `requestId`. + +### Why can't drivers split bulk write commands to maximize retryability? + +In [Supported Write Operations](#supported-write-operations), the spec prohibits drivers from altering existing logic +for splits `bulkWrite()`'s `requests` parameter into write commands in an attempt to segregate unsupported, +multi-document write operations and maximize retryability for other, supported write operations. The reasoning behind +this prohibition is that such behavior would conflict with a primary goal of the bulk API in reducing the number of +command round-trips to the server. + +### retryWrites originally defaulted to false, why does it now default to true? + +Since the initial release of retryable writes in MongoDB 3.6 testing showed that the overhead for supported operations +was sufficiently small that there was no risk in changing the default. Additionally, the fact that some operations +continue to be unsupported for retryable writes (updateMany and deleteMany) does not seem to pose a problem in practice. + +### Why do drivers have to parse errmsg to determine storage engine support? + +There is no reliable way to determine the storage engine in use for shards in a sharded cluster, and replica sets (and +shards) can have mixed deployments using different storage engines on different members. This is especially true when a +replica set or sharded cluster is being upgraded from one storage engine to another. This could be common when upgrading +to MongoDB 4.2, where MMAPv1 is no longer supported. + +The server returns error code 20 (IllegalOperation) when the storage engine doesn't support document-level locking and +txnNumbers. Error code 20 is used for a large number of different error cases in the server so we need some other way to +differentiate this error case from any other. The error code and errmsg are the same in MongoDB 3.6 and 4.0, and the +same from a replica set or sharded cluster (mongos just forwards the error from the shard's replica set). + +### Why does the driver only add the RetryableWriteError label to errors that occur on a MongoClient with retryWrites set to true? + +The driver does this to maintain consistency with the MongoDB server. Servers that support the RetryableWriteError label +(MongoDB version 4.4 and newer) only add the label to an error when the client has added a txnNumber to the command, +which only happens when the retryWrites option is true on the client. For the driver to add the label even if +retryWrites is not true would be inconsistent with the server and potentially confusing to developers. + +## Changelog + +- 2024-05-02: Migrated from reStructuredText to Markdown. + +- 2024-04-29: Fix the link to the Driver Sessions spec. + +- 2024-01-16: Do not use `writeConcernError.code` in pre-4.4 mongos response to\ + determine retryability. Do not use + `writeErrors[].code` in pre-4.4 server responses to determine retryability. + +- 2023-12-06: Clarify that writes are not retried within transactions. + +- 2023-12-05: Add that any server information associated with retryable\ + exceptions MUST reflect the originating server, + even in the presence of retries. + +- 2023-10-02: When CSOT is not enabled, one retry attempt occurs. + +- 2023-08-26: Require that in a sharded cluster the server on which the\ + operation failed MUST be provided to the server + selection mechanism as a deprioritized server. + +- 2022-11-17: Add logic for persisting "currentError" as "previousError" on first\ + retry attempt, avoiding raising + "null" errors. + +- 2022-11-09: CLAM must apply both events and log messages. + +- 2022-10-18: When CSOT is enabled multiple retry attempts may occur. + +- 2022-10-05: Remove spec front matter and reformat changelog. + +- 2022-01-25: Note that drivers should retry handshake network failures. + +- 2021-11-02: Clarify that error labels are only specified in a top-level field\ + of an error. + +- 2021-04-26: Replaced deprecated terminology + +- 2021-03-24: Require that PoolClearedErrors be retried + +- 2020-09-01: State the the driver should only add the RetryableWriteError label\ + to network errors when connected to a + 4.4+ server. + +- 2020-02-25: State that the driver should only add the RetryableWriteError label\ + when retryWrites is on, and make it + clear that mongos will sometimes perform internal retries and not return the RetryableWriteError label. + +- 2020-02-10: Remove redundant content in Tests section. + +- 2020-01-14: Add ExceededTimeLimit to the list of error codes that should\ + receive a RetryableWriteError label. + +- 2019-10-21: Change the definition of "retryable write" to be based on the\ + RetryableWriteError label. Stop requiring + drivers to parse errmsg to categorize retryable errors for pre-4.4 servers. + +- 2019-07-30: Drivers must rewrite error messages for error code 20 when\ + txnNumber is not supported by the storage + engine. + +- 2019-06-07: Mention `$merge` stage for aggregate alongside `$out` + +- 2019-05-29: Renamed InterruptedDueToStepDown to InterruptedDueToReplStateChange + +- 2019-03-06: retryWrites now defaults to true. + +- 2019-03-05: Prohibit resending wire protocol messages if doing so would violate\ + rules for gossipping the cluster + time. + +- 2018-06-07: WriteConcernFailed is not a retryable error code. + +- 2018-04-25: Evaluate retryable eligibility of bulkWrite() commands individually. + +- 2018-03-14: Clarify that retryable writes may fail with a FCV 3.4 shard. + +- 2017-11-02: Drivers should not raise errors if selected server does not support\ + retryable writes and instead fall + back to non-retryable behavior. In addition to wire protocol version, drivers may check for + `logicalSessionTimeoutMinutes` to determine if a server supports sessions and retryable writes. + +- 2017-10-26: Errors when retrying may be raised instead of the original error\ + provided they allow the user to infer + that an attempt was made. + +- 2017-10-23: Drivers must document operations that support retryability. + +- 2017-10-23: Raise the original retryable error if server selection or wire\ + protocol checks fail during the retry + attempt. Encourage drivers to provide intermediary write results after an unrecoverable failure during a bulk write. + +- 2017-10-18: Standalone servers do not support retryable writes. + +- 2017-10-18: Also retry writes after a "not writable primary" error. + +- 2017-10-08: Renamed `txnNum` to `txnNumber` and noted that it must be a\ + 64-bit integer (BSON type 0x12). + +- 2017-08-25: Drivers will maintain an allow list so that only supported write\ + operations may be retried. Transaction + IDs will not be included in unsupported write commands, irrespective of the `retryWrites` option. + +- 2017-08-18: `retryWrites` is now a MongoClient option. diff --git a/source/retryable-writes/retryable-writes.rst b/source/retryable-writes/retryable-writes.rst new file mode 100644 index 0000000000..9d1c8d2ef5 --- /dev/null +++ b/source/retryable-writes/retryable-writes.rst @@ -0,0 +1,4 @@ + +.. note:: + This specification has been converted to Markdown and renamed to + `retryable-writes.md `_. diff --git a/source/retryable-writes/tests/README.md b/source/retryable-writes/tests/README.md index e0330ff3df..151b26181f 100644 --- a/source/retryable-writes/tests/README.md +++ b/source/retryable-writes/tests/README.md @@ -1,394 +1,122 @@ -===================== -Retryable Write Tests -===================== +# Retryable Write Tests -.. contents:: +______________________________________________________________________ ----- +## Introduction -Introduction -============ +The YAML and JSON files in this directory are platform-independent tests meant to exercise a driver's implementation of +retryable writes. These tests utilize the [Unified Test Format](../../unified-test-format/unified-test-format.md). -The YAML and JSON files in this directory are platform-independent tests meant -to exercise a driver's implementation of retryable writes. These tests utilize -the [Unified Test Format](../../unified-test-format/unified-test-format.md). +Several prose tests, which are not easily expressed in YAML, are also presented in this file. Those tests will need to +be manually implemented by each driver. -Several prose tests, which are not easily expressed in YAML, are also presented -in this file. Those tests will need to be manually implemented by each driver. +Tests will require a MongoClient created with options defined in the tests. Integration tests will require a running +MongoDB cluster with server versions 3.6.0 or later. The `{setFeatureCompatibilityVersion: 3.6}` admin command will also +need to have been executed to enable support for retryable writes on the cluster. Some tests may have more stringent +version requirements depending on the fail points used. -Tests will require a MongoClient created with options defined in the tests. -Integration tests will require a running MongoDB cluster with server versions -3.6.0 or later. The ``{setFeatureCompatibilityVersion: 3.6}`` admin command -will also need to have been executed to enable support for retryable writes on -the cluster. Some tests may have more stringent version requirements depending -on the fail points used. +## Use as Integration Tests -Use as Integration Tests -======================== - -Integration tests are expressed in YAML and can be run against a replica set or -sharded cluster as denoted by the top-level ``runOn`` field. Tests that rely on -the ``onPrimaryTransactionalWrite`` fail point cannot be run against a sharded +Integration tests are expressed in YAML and can be run against a replica set or sharded cluster as denoted by the +top-level `runOn` field. Tests that rely on the `onPrimaryTransactionalWrite` fail point cannot be run against a sharded cluster because the fail point is not supported by mongos. The tests exercise the following scenarios: - Single-statement write operations - - - Each test expecting a write result will encounter at-most one network error - for the write command. Retry attempts should return without error and allow - operation to succeed. Observation of the collection state will assert that - the write occurred at-most once. - - - Each test expecting an error will encounter successive network errors for - the write command. Observation of the collection state will assert that the - write was never committed on the server. - + - Each test expecting a write result will encounter at-most one network error for the write command. Retry attempts + should return without error and allow operation to succeed. Observation of the collection state will assert that the + write occurred at-most once. + - Each test expecting an error will encounter successive network errors for the write command. Observation of the + collection state will assert that the write was never committed on the server. - Multi-statement write operations - - - Each test expecting a write result will encounter at-most one network error - for some write command(s) in the batch. Retry attempts should return without - error and allow the batch to ultimately succeed. Observation of the - collection state will assert that each write occurred at-most once. - - - Each test expecting an error will encounter successive network errors for - some write command in the batch. The batch will ultimately fail with an - error, but observation of the collection state will assert that the failing - write was never committed on the server. We may observe that earlier writes - in the batch occurred at-most once. - -We cannot test a scenario where the first and second attempts both encounter -network errors but the write does actually commit during one of those attempts. -This is because (1) the fail point only triggers when a write would be committed -and (2) the skip and times options are mutually exclusive. That said, such a -test would mainly assert the server's correctness for at-most once semantics and -is not essential to assert driver correctness. - -Split Batch Tests -================= - -The YAML tests specify bulk write operations that are split by command type -(e.g. sequence of insert, update, and delete commands). Multi-statement write -operations may also be split due to ``maxWriteBatchSize``, -``maxBsonObjectSize``, or ``maxMessageSizeBytes``. - -For instance, an insertMany operation with five 10 MiB documents executed using -OP_MSG payload type 0 (i.e. entire command in one document) would be split into -five insert commands in order to respect the 16 MiB ``maxBsonObjectSize`` limit. -The same insertMany operation executed using OP_MSG payload type 1 (i.e. command -arguments pulled out into a separate payload vector) would be split into two -insert commands in order to respect the 48 MB ``maxMessageSizeBytes`` limit. - -Noting when a driver might split operations, the ``onPrimaryTransactionalWrite`` -fail point's ``skip`` option may be used to control when the fail point first -triggers. Once triggered, the fail point will transition to the ``alwaysOn`` -state until disabled. Driver authors should also note that the server attempts -to process all documents in a single insert command within a single commit (i.e. -one insert command with five documents may only trigger the fail point once). -This behavior is unique to insert commands (each statement in an update and -delete command is processed independently). - -If testing an insert that is split into two commands, a ``skip`` of one will -allow the fail point to trigger on the second insert command (because all -documents in the first command will be processed in the same commit). When -testing an update or delete that is split into two commands, the ``skip`` should -be set to the number of statements in the first command to allow the fail point -to trigger on the second command. - -Command Construction Tests -========================== - -Drivers should also assert that command documents are properly constructed with -or without a transaction ID, depending on whether the write operation is -supported. `Command Logging and Monitoring`_ may be used to check for the presence of a -``txnNumber`` field in the command document. Note that command documents may -always include an ``lsid`` field per the `Driver Session`_ specification. - -.. _Command Logging and Monitoring: ../../command-logging-and-monitoring/command-logging-and-monitoring.rst -.. _Driver Session: ../../sessions/driver-sessions.rst + - Each test expecting a write result will encounter at-most one network error for some write command(s) in the batch. + Retry attempts should return without error and allow the batch to ultimately succeed. Observation of the collection + state will assert that each write occurred at-most once. + - Each test expecting an error will encounter successive network errors for some write command in the batch. The batch + will ultimately fail with an error, but observation of the collection state will assert that the failing write was + never committed on the server. We may observe that earlier writes in the batch occurred at-most once. + +We cannot test a scenario where the first and second attempts both encounter network errors but the write does actually +commit during one of those attempts. This is because (1) the fail point only triggers when a write would be committed +and (2) the skip and times options are mutually exclusive. That said, such a test would mainly assert the server's +correctness for at-most once semantics and is not essential to assert driver correctness. + +## Split Batch Tests + +The YAML tests specify bulk write operations that are split by command type (e.g. sequence of insert, update, and delete +commands). Multi-statement write operations may also be split due to `maxWriteBatchSize`, `maxBsonObjectSize`, or +`maxMessageSizeBytes`. + +For instance, an insertMany operation with five 10 MiB documents executed using OP_MSG payload type 0 (i.e. entire +command in one document) would be split into five insert commands in order to respect the 16 MiB `maxBsonObjectSize` +limit. The same insertMany operation executed using OP_MSG payload type 1 (i.e. command arguments pulled out into a +separate payload vector) would be split into two insert commands in order to respect the 48 MB `maxMessageSizeBytes` +limit. + +Noting when a driver might split operations, the `onPrimaryTransactionalWrite` fail point's `skip` option may be used to +control when the fail point first triggers. Once triggered, the fail point will transition to the `alwaysOn` state until +disabled. Driver authors should also note that the server attempts to process all documents in a single insert command +within a single commit (i.e. one insert command with five documents may only trigger the fail point once). This behavior +is unique to insert commands (each statement in an update and delete command is processed independently). + +If testing an insert that is split into two commands, a `skip` of one will allow the fail point to trigger on the second +insert command (because all documents in the first command will be processed in the same commit). When testing an update +or delete that is split into two commands, the `skip` should be set to the number of statements in the first command to +allow the fail point to trigger on the second command. + +## Command Construction Tests + +Drivers should also assert that command documents are properly constructed with or without a transaction ID, depending +on whether the write operation is supported. +[Command Logging and Monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.rst) may be used to +check for the presence of a `txnNumber` field in the command document. Note that command documents may always include an +`lsid` field per the [Driver Session](../../sessions/driver-sessions.rst) specification. These tests may be run against both a replica set and shard cluster. -Drivers should test that transaction IDs are never included in commands for -unsupported write operations: - -* Write commands with unacknowledged write concerns (e.g. ``{w: 0}``) - -* Unsupported single-statement write operations - - - ``updateMany()`` - - ``deleteMany()`` - -* Unsupported multi-statement write operations - - - ``bulkWrite()`` that includes ``UpdateMany`` or ``DeleteMany`` - -* Unsupported write commands - - - ``aggregate`` with write stage (e.g. ``$out``, ``$merge``) - -Drivers should test that transactions IDs are always included in commands for -supported write operations: - -* Supported single-statement write operations - - - ``insertOne()`` - - ``updateOne()`` - - ``replaceOne()`` - - ``deleteOne()`` - - ``findOneAndDelete()`` - - ``findOneAndReplace()`` - - ``findOneAndUpdate()`` - -* Supported multi-statement write operations - - - ``insertMany()`` with ``ordered=true`` - - ``insertMany()`` with ``ordered=false`` - - ``bulkWrite()`` with ``ordered=true`` (no ``UpdateMany`` or ``DeleteMany``) - - ``bulkWrite()`` with ``ordered=false`` (no ``UpdateMany`` or ``DeleteMany``) - -Prose Tests -=========== - -The following tests ensure that retryable writes work properly with replica sets -and sharded clusters. - -#. Test that retryable writes raise an exception when using the MMAPv1 storage - engine. For this test, execute a write operation, such as ``insertOne``, - which should generate an exception. Assert that the error message is the - replacement error message:: - - This MongoDB deployment does not support retryable writes. Please add - retryWrites=false to your connection string. +Drivers should test that transaction IDs are never included in commands for unsupported write operations: + +- Write commands with unacknowledged write concerns (e.g. `{w: 0}`) +- Unsupported single-statement write operations + - `updateMany()` + - `deleteMany()` +- Unsupported multi-statement write operations + - `bulkWrite()` that includes `UpdateMany` or `DeleteMany` +- Unsupported write commands + - `aggregate` with write stage (e.g. `$out`, `$merge`) + +Drivers should test that transactions IDs are always included in commands for supported write operations: + +- Supported single-statement write operations + - `insertOne()` + - `updateOne()` + - `replaceOne()` + - `deleteOne()` + - `findOneAndDelete()` + - `findOneAndReplace()` + - `findOneAndUpdate()` +- Supported multi-statement write operations + - `insertMany()` with `ordered=true` + - `insertMany()` with `ordered=false` + - `bulkWrite()` with `ordered=true` (no `UpdateMany` or `DeleteMany`) + - `bulkWrite()` with `ordered=false` (no `UpdateMany` or `DeleteMany`) + +## Prose Tests + +The following tests ensure that retryable writes work properly with replica sets and sharded clusters. + +1. Test that retryable writes raise an exception when using the MMAPv1 storage engine. For this test, execute a write + operation, such as `insertOne`, which should generate an exception. Assert that the error message is the replacement + error message: + + ``` + This MongoDB deployment does not support retryable writes. Please add + retryWrites=false to your connection string. + ``` and the error code is 20. - **Note**: Drivers that rely on ``serverStatus`` to determine the storage engine - in use MAY skip this test for sharded clusters, since ``mongos`` does not report - this information in its ``serverStatus`` response. - -#. Test that drivers properly retry after encountering PoolClearedErrors. This - test MUST be implemented by any driver that implements the CMAP - specification. - - This test requires MongoDB 4.3.4+ for both the ``errorLabels`` and - ``blockConnection`` fail point options. - - 1. Create a client with maxPoolSize=1 and retryWrites=true. If testing - against a sharded deployment, be sure to connect to only a single mongos. - - 2. Enable the following failpoint:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 91, - blockConnection: true, - blockTimeMS: 1000, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Start two threads and attempt to perform an ``insertOne`` simultaneously on both. - - 4. Verify that both ``insertOne`` attempts succeed. - - 5. Via CMAP monitoring, assert that the first check out succeeds. - - 6. Via CMAP monitoring, assert that a PoolClearedEvent is then emitted. - - 7. Via CMAP monitoring, assert that the second check out then fails due to a - connection error. - - 8. Via Command Monitoring, assert that exactly three ``insert`` - CommandStartedEvents were observed in total. - - 9. Disable the failpoint. - -#. Test that drivers return the original error after encountering a - WriteConcernError with a RetryableWriteError label. This test MUST - - 1. be implemented by any driver that implements the Command Monitoring - specification, - - 2. only run against replica sets as mongos does not propagate the - NoWritesPerformed label to the drivers. - - 3. be run against server versions 6.0 and above. - - Additionally, this test requires drivers to set a fail point after an - ``insertOne`` operation but before the subsequent retry. Drivers that are - unable to set a failCommand after the CommandSucceededEvent SHOULD use - mocking or write a unit test to cover the same sequence of events. - - - 1. Create a client with ``retryWrites=true``. - - 2. Configure a fail point with error code ``91`` (ShutdownInProgress):: - - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorLabels: ["RetryableWriteError"], - writeConcernError: { code: 91 } - } - } - - 3. Via the command monitoring CommandSucceededEvent, configure a fail point - with error code ``10107`` (NotWritablePrimary) and a NoWritesPerformed - label:: - - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorCode: 10107, - errorLabels: ["RetryableWriteError", "NoWritesPerformed"] - } - } - - Drivers SHOULD only configure the ``10107`` fail point command if the the - succeeded event is for the ``91`` error configured in step 2. - - 4. Attempt an ``insertOne`` operation on any record for any database and - collection. For the resulting error, assert that the associated error code - is ``91``. - - 5. Disable the fail point:: - - { - configureFailPoint: "failCommand", - mode: "off" - } - -#. Test that in a sharded cluster writes are retried on a different mongos when - one is available. - - This test MUST be executed against a sharded cluster that has at least two - mongos instances, supports ``retryWrites=true``, has enabled the - ``configureFailPoint`` command, and supports the ``errorLabels`` field - (MongoDB 4.3.1+). - - Note: this test cannot reliably distinguish "retry on a different mongos due - to server deprioritization" (the behavior intended to be tested) from "retry - on a different mongos due to normal SDAM randomized suitable server - selection". Verify relevant code paths are correctly executed by the tests - using external means such as a logging, debugger, code coverage tool, etc. - - 1. Create two clients ``s0`` and ``s1`` that each connect to a single mongos - from the sharded cluster. They must not connect to the same mongos. - - 2. Configure the following fail point for both ``s0`` and ``s1``:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Create a client ``client`` with ``retryWrites=true`` that connects to the - cluster using the same two mongoses as ``s0`` and ``s1``. - - 4. Enable failed command event monitoring for ``client``. - - 5. Execute an ``insert`` command with ``client``. Assert that the command - failed. - - 6. Assert that two failed command events occurred. Assert that the failed - command events occurred on different mongoses. - - 7. Disable the fail points on both ``s0`` and ``s1``. - -#. Test that in a sharded cluster writes are retried on the same mongos when no - others are available. - - This test MUST be executed against a sharded cluster that supports - ``retryWrites=true``, has enabled the ``configureFailPoint`` command, and - supports the ``errorLabels`` field (MongoDB 4.3.1+). - - Note: this test cannot reliably distinguish "retry on a different mongos due - to server deprioritization" (the behavior intended to be tested) from "retry - on a different mongos due to normal SDAM behavior of randomized suitable - server selection". Verify relevant code paths are correctly executed by the - tests using external means such as a logging, debugger, code coverage tool, - etc. - - 1. Create a client ``s0`` that connects to a single mongos from the cluster. - - 2. Configure the following fail point for ``s0``:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Create a client ``client`` with ``directConnection=false`` (when not set by - default) and ``retryWrites=true`` that connects to the cluster using the - same single mongos as ``s0``. - - 4. Enable succeeded and failed command event monitoring for ``client``. - - 5. Execute an ``insert`` command with ``client``. Assert that the command - succeeded. - - 6. Assert that exactly one failed command event and one succeeded command - event occurred. Assert that both events occurred on the same mongos. - - 7. Disable the fail point on ``s0``. - -Changelog -========= - -:2024-02-27: Convert legacy retryable writes tests to unified format. - -:2024-02-21: Update prose test 4 and 5 to workaround SDAM behavior preventing - execution of deprioritization code paths. - -:2024-01-05: Fix typo in prose test title. - -:2024-01-03: Note server version requirements for fail point options and revise - tests to specify the ``errorLabels`` option at the top-level - instead of within ``writeConcernError``. - -:2023-08-26: Add prose tests for retrying in a sharded cluster. - -:2022-08-30: Add prose test verifying correct error handling for errors with - the NoWritesPerformed label, which is to return the original - error. - -:2022-04-22: Clarifications to ``serverless`` and ``useMultipleMongoses``. - -:2021-08-27: Add ``serverless`` to ``runOn``. Clarify behavior of - ``useMultipleMongoses`` for ``LoadBalanced`` topologies. - -:2021-04-23: Add ``load-balanced`` to test topology requirements. - -:2021-03-24: Add prose test verifying ``PoolClearedErrors`` are retried. - -:2019-10-21: Add ``errorLabelsContain`` and ``errorLabelsContain`` fields to - ``result`` - -:2019-08-07: Add Prose Tests section - -:2019-06-07: Mention $merge stage for aggregate alongside $out - -:2019-03-01: Add top-level ``runOn`` field to denote server version and/or - topology requirements requirements for the test file. Removes the - ``minServerVersion`` and ``maxServerVersion`` top-level fields, - which are now expressed within ``runOn`` elements. - - Add test-level ``useMultipleMongoses`` field. + [!NOTE] + storage engine in use MAY skip this test for sharded clusters, since `mongos` does not report this information in its + `serverStatus` response. diff --git a/source/retryable-writes/tests/README.rst b/source/retryable-writes/tests/README.rst deleted file mode 100644 index e0330ff3df..0000000000 --- a/source/retryable-writes/tests/README.rst +++ /dev/null @@ -1,394 +0,0 @@ -===================== -Retryable Write Tests -===================== - -.. contents:: - ----- - -Introduction -============ - -The YAML and JSON files in this directory are platform-independent tests meant -to exercise a driver's implementation of retryable writes. These tests utilize -the [Unified Test Format](../../unified-test-format/unified-test-format.md). - -Several prose tests, which are not easily expressed in YAML, are also presented -in this file. Those tests will need to be manually implemented by each driver. - -Tests will require a MongoClient created with options defined in the tests. -Integration tests will require a running MongoDB cluster with server versions -3.6.0 or later. The ``{setFeatureCompatibilityVersion: 3.6}`` admin command -will also need to have been executed to enable support for retryable writes on -the cluster. Some tests may have more stringent version requirements depending -on the fail points used. - -Use as Integration Tests -======================== - -Integration tests are expressed in YAML and can be run against a replica set or -sharded cluster as denoted by the top-level ``runOn`` field. Tests that rely on -the ``onPrimaryTransactionalWrite`` fail point cannot be run against a sharded -cluster because the fail point is not supported by mongos. - -The tests exercise the following scenarios: - -- Single-statement write operations - - - Each test expecting a write result will encounter at-most one network error - for the write command. Retry attempts should return without error and allow - operation to succeed. Observation of the collection state will assert that - the write occurred at-most once. - - - Each test expecting an error will encounter successive network errors for - the write command. Observation of the collection state will assert that the - write was never committed on the server. - -- Multi-statement write operations - - - Each test expecting a write result will encounter at-most one network error - for some write command(s) in the batch. Retry attempts should return without - error and allow the batch to ultimately succeed. Observation of the - collection state will assert that each write occurred at-most once. - - - Each test expecting an error will encounter successive network errors for - some write command in the batch. The batch will ultimately fail with an - error, but observation of the collection state will assert that the failing - write was never committed on the server. We may observe that earlier writes - in the batch occurred at-most once. - -We cannot test a scenario where the first and second attempts both encounter -network errors but the write does actually commit during one of those attempts. -This is because (1) the fail point only triggers when a write would be committed -and (2) the skip and times options are mutually exclusive. That said, such a -test would mainly assert the server's correctness for at-most once semantics and -is not essential to assert driver correctness. - -Split Batch Tests -================= - -The YAML tests specify bulk write operations that are split by command type -(e.g. sequence of insert, update, and delete commands). Multi-statement write -operations may also be split due to ``maxWriteBatchSize``, -``maxBsonObjectSize``, or ``maxMessageSizeBytes``. - -For instance, an insertMany operation with five 10 MiB documents executed using -OP_MSG payload type 0 (i.e. entire command in one document) would be split into -five insert commands in order to respect the 16 MiB ``maxBsonObjectSize`` limit. -The same insertMany operation executed using OP_MSG payload type 1 (i.e. command -arguments pulled out into a separate payload vector) would be split into two -insert commands in order to respect the 48 MB ``maxMessageSizeBytes`` limit. - -Noting when a driver might split operations, the ``onPrimaryTransactionalWrite`` -fail point's ``skip`` option may be used to control when the fail point first -triggers. Once triggered, the fail point will transition to the ``alwaysOn`` -state until disabled. Driver authors should also note that the server attempts -to process all documents in a single insert command within a single commit (i.e. -one insert command with five documents may only trigger the fail point once). -This behavior is unique to insert commands (each statement in an update and -delete command is processed independently). - -If testing an insert that is split into two commands, a ``skip`` of one will -allow the fail point to trigger on the second insert command (because all -documents in the first command will be processed in the same commit). When -testing an update or delete that is split into two commands, the ``skip`` should -be set to the number of statements in the first command to allow the fail point -to trigger on the second command. - -Command Construction Tests -========================== - -Drivers should also assert that command documents are properly constructed with -or without a transaction ID, depending on whether the write operation is -supported. `Command Logging and Monitoring`_ may be used to check for the presence of a -``txnNumber`` field in the command document. Note that command documents may -always include an ``lsid`` field per the `Driver Session`_ specification. - -.. _Command Logging and Monitoring: ../../command-logging-and-monitoring/command-logging-and-monitoring.rst -.. _Driver Session: ../../sessions/driver-sessions.rst - -These tests may be run against both a replica set and shard cluster. - -Drivers should test that transaction IDs are never included in commands for -unsupported write operations: - -* Write commands with unacknowledged write concerns (e.g. ``{w: 0}``) - -* Unsupported single-statement write operations - - - ``updateMany()`` - - ``deleteMany()`` - -* Unsupported multi-statement write operations - - - ``bulkWrite()`` that includes ``UpdateMany`` or ``DeleteMany`` - -* Unsupported write commands - - - ``aggregate`` with write stage (e.g. ``$out``, ``$merge``) - -Drivers should test that transactions IDs are always included in commands for -supported write operations: - -* Supported single-statement write operations - - - ``insertOne()`` - - ``updateOne()`` - - ``replaceOne()`` - - ``deleteOne()`` - - ``findOneAndDelete()`` - - ``findOneAndReplace()`` - - ``findOneAndUpdate()`` - -* Supported multi-statement write operations - - - ``insertMany()`` with ``ordered=true`` - - ``insertMany()`` with ``ordered=false`` - - ``bulkWrite()`` with ``ordered=true`` (no ``UpdateMany`` or ``DeleteMany``) - - ``bulkWrite()`` with ``ordered=false`` (no ``UpdateMany`` or ``DeleteMany``) - -Prose Tests -=========== - -The following tests ensure that retryable writes work properly with replica sets -and sharded clusters. - -#. Test that retryable writes raise an exception when using the MMAPv1 storage - engine. For this test, execute a write operation, such as ``insertOne``, - which should generate an exception. Assert that the error message is the - replacement error message:: - - This MongoDB deployment does not support retryable writes. Please add - retryWrites=false to your connection string. - - and the error code is 20. - - **Note**: Drivers that rely on ``serverStatus`` to determine the storage engine - in use MAY skip this test for sharded clusters, since ``mongos`` does not report - this information in its ``serverStatus`` response. - -#. Test that drivers properly retry after encountering PoolClearedErrors. This - test MUST be implemented by any driver that implements the CMAP - specification. - - This test requires MongoDB 4.3.4+ for both the ``errorLabels`` and - ``blockConnection`` fail point options. - - 1. Create a client with maxPoolSize=1 and retryWrites=true. If testing - against a sharded deployment, be sure to connect to only a single mongos. - - 2. Enable the following failpoint:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 91, - blockConnection: true, - blockTimeMS: 1000, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Start two threads and attempt to perform an ``insertOne`` simultaneously on both. - - 4. Verify that both ``insertOne`` attempts succeed. - - 5. Via CMAP monitoring, assert that the first check out succeeds. - - 6. Via CMAP monitoring, assert that a PoolClearedEvent is then emitted. - - 7. Via CMAP monitoring, assert that the second check out then fails due to a - connection error. - - 8. Via Command Monitoring, assert that exactly three ``insert`` - CommandStartedEvents were observed in total. - - 9. Disable the failpoint. - -#. Test that drivers return the original error after encountering a - WriteConcernError with a RetryableWriteError label. This test MUST - - 1. be implemented by any driver that implements the Command Monitoring - specification, - - 2. only run against replica sets as mongos does not propagate the - NoWritesPerformed label to the drivers. - - 3. be run against server versions 6.0 and above. - - Additionally, this test requires drivers to set a fail point after an - ``insertOne`` operation but before the subsequent retry. Drivers that are - unable to set a failCommand after the CommandSucceededEvent SHOULD use - mocking or write a unit test to cover the same sequence of events. - - - 1. Create a client with ``retryWrites=true``. - - 2. Configure a fail point with error code ``91`` (ShutdownInProgress):: - - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorLabels: ["RetryableWriteError"], - writeConcernError: { code: 91 } - } - } - - 3. Via the command monitoring CommandSucceededEvent, configure a fail point - with error code ``10107`` (NotWritablePrimary) and a NoWritesPerformed - label:: - - { - configureFailPoint: "failCommand", - mode: {times: 1}, - data: { - failCommands: ["insert"], - errorCode: 10107, - errorLabels: ["RetryableWriteError", "NoWritesPerformed"] - } - } - - Drivers SHOULD only configure the ``10107`` fail point command if the the - succeeded event is for the ``91`` error configured in step 2. - - 4. Attempt an ``insertOne`` operation on any record for any database and - collection. For the resulting error, assert that the associated error code - is ``91``. - - 5. Disable the fail point:: - - { - configureFailPoint: "failCommand", - mode: "off" - } - -#. Test that in a sharded cluster writes are retried on a different mongos when - one is available. - - This test MUST be executed against a sharded cluster that has at least two - mongos instances, supports ``retryWrites=true``, has enabled the - ``configureFailPoint`` command, and supports the ``errorLabels`` field - (MongoDB 4.3.1+). - - Note: this test cannot reliably distinguish "retry on a different mongos due - to server deprioritization" (the behavior intended to be tested) from "retry - on a different mongos due to normal SDAM randomized suitable server - selection". Verify relevant code paths are correctly executed by the tests - using external means such as a logging, debugger, code coverage tool, etc. - - 1. Create two clients ``s0`` and ``s1`` that each connect to a single mongos - from the sharded cluster. They must not connect to the same mongos. - - 2. Configure the following fail point for both ``s0`` and ``s1``:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Create a client ``client`` with ``retryWrites=true`` that connects to the - cluster using the same two mongoses as ``s0`` and ``s1``. - - 4. Enable failed command event monitoring for ``client``. - - 5. Execute an ``insert`` command with ``client``. Assert that the command - failed. - - 6. Assert that two failed command events occurred. Assert that the failed - command events occurred on different mongoses. - - 7. Disable the fail points on both ``s0`` and ``s1``. - -#. Test that in a sharded cluster writes are retried on the same mongos when no - others are available. - - This test MUST be executed against a sharded cluster that supports - ``retryWrites=true``, has enabled the ``configureFailPoint`` command, and - supports the ``errorLabels`` field (MongoDB 4.3.1+). - - Note: this test cannot reliably distinguish "retry on a different mongos due - to server deprioritization" (the behavior intended to be tested) from "retry - on a different mongos due to normal SDAM behavior of randomized suitable - server selection". Verify relevant code paths are correctly executed by the - tests using external means such as a logging, debugger, code coverage tool, - etc. - - 1. Create a client ``s0`` that connects to a single mongos from the cluster. - - 2. Configure the following fail point for ``s0``:: - - { - configureFailPoint: "failCommand", - mode: { times: 1 }, - data: { - failCommands: ["insert"], - errorCode: 6, - errorLabels: ["RetryableWriteError"] - } - } - - 3. Create a client ``client`` with ``directConnection=false`` (when not set by - default) and ``retryWrites=true`` that connects to the cluster using the - same single mongos as ``s0``. - - 4. Enable succeeded and failed command event monitoring for ``client``. - - 5. Execute an ``insert`` command with ``client``. Assert that the command - succeeded. - - 6. Assert that exactly one failed command event and one succeeded command - event occurred. Assert that both events occurred on the same mongos. - - 7. Disable the fail point on ``s0``. - -Changelog -========= - -:2024-02-27: Convert legacy retryable writes tests to unified format. - -:2024-02-21: Update prose test 4 and 5 to workaround SDAM behavior preventing - execution of deprioritization code paths. - -:2024-01-05: Fix typo in prose test title. - -:2024-01-03: Note server version requirements for fail point options and revise - tests to specify the ``errorLabels`` option at the top-level - instead of within ``writeConcernError``. - -:2023-08-26: Add prose tests for retrying in a sharded cluster. - -:2022-08-30: Add prose test verifying correct error handling for errors with - the NoWritesPerformed label, which is to return the original - error. - -:2022-04-22: Clarifications to ``serverless`` and ``useMultipleMongoses``. - -:2021-08-27: Add ``serverless`` to ``runOn``. Clarify behavior of - ``useMultipleMongoses`` for ``LoadBalanced`` topologies. - -:2021-04-23: Add ``load-balanced`` to test topology requirements. - -:2021-03-24: Add prose test verifying ``PoolClearedErrors`` are retried. - -:2019-10-21: Add ``errorLabelsContain`` and ``errorLabelsContain`` fields to - ``result`` - -:2019-08-07: Add Prose Tests section - -:2019-06-07: Mention $merge stage for aggregate alongside $out - -:2019-03-01: Add top-level ``runOn`` field to denote server version and/or - topology requirements requirements for the test file. Removes the - ``minServerVersion`` and ``maxServerVersion`` top-level fields, - which are now expressed within ``runOn`` elements. - - Add test-level ``useMultipleMongoses`` field. diff --git a/source/run-command/run-command.rst b/source/run-command/run-command.rst index 5cea141368..a51f2b0261 100644 --- a/source/run-command/run-command.rst +++ b/source/run-command/run-command.rst @@ -174,7 +174,7 @@ All commands executed via RunCommand are non-retryable operations. Drivers MUST NOT inspect the command to determine if it is a write and MUST NOT attach a ``txnNumber``. * See Retryable Reads' section on `Unsupported Read Operations <../retryable-reads/retryable-reads.md#unsupported-read-operations>`_ -* See Retryable Writes' section on `Behavioral Changes for Write Commands `_ +* See Retryable Writes' section on `Behavioral Changes for Write Commands <../retryable-writes/retryable-writes.md#behavioral-changes-for-write-commands>`_ Stable API """""""""" diff --git a/source/transactions-convenient-api/transactions-convenient-api.rst b/source/transactions-convenient-api/transactions-convenient-api.rst index a2003bf5ce..82f1136193 100644 --- a/source/transactions-convenient-api/transactions-convenient-api.rst +++ b/source/transactions-convenient-api/transactions-convenient-api.rst @@ -414,7 +414,7 @@ retry ``commitTransaction`` if it fails with any error bearing the `UnknownTransactionCommitResult`_ error label. This label is applied for the the following errors: -.. _Retryable Writes: ../retryable-writes/retryable-writes.rst#terms +.. _Retryable Writes: ../retryable-writes/retryable-writes.md#terms .. _UnknownTransactionCommitResult: ../transactions/transactions.md#unknowntransactioncommitresult diff --git a/source/transactions/transactions.md b/source/transactions/transactions.md index 4fa5c6c7fc..76745b59bb 100644 --- a/source/transactions/transactions.md +++ b/source/transactions/transactions.md @@ -24,7 +24,7 @@ The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SH ### **Terms** This specification uses the terms defined in the [Driver Sessions Specification](../sessions/driver-sessions.rst) and -[Retryable Writes Specification](../retryable-writes/retryable-writes.rst). Additional terms are defined below. +[Retryable Writes Specification](../retryable-writes/retryable-writes.md). Additional terms are defined below. #### Resource Management Block @@ -46,7 +46,7 @@ including (but not limited to) creating, updating, or deleting databases, collec #### Retryable Error -An error considered retryable by the [Retryable Writes Specification](../retryable-writes/retryable-writes.rst). +An error considered retryable by the [Retryable Writes Specification](../retryable-writes/retryable-writes.md). #### Command Error @@ -375,7 +375,7 @@ state. When the session is in the "starting transaction" state, meaning, no oper transaction, drivers MUST NOT run the abortTransaction command. abortTransaction is a retryable write command. Drivers MUST retry after abortTransaction fails with a retryable error -according to the [Retryable Writes Specification](../retryable-writes/retryable-writes.rst), including a handshake +according to the [Retryable Writes Specification](../retryable-writes/retryable-writes.md), including a handshake network error, regardless of whether retryWrites is set on the MongoClient or not. If the operation times out or fails with a non-retryable error, drivers MUST ignore all errors from the abortTransaction @@ -553,7 +553,7 @@ been enabled on the MongoClient. Drivers MUST retry the commitTransaction and abortTransaction commands even when retryWrites has been disabled on the MongoClient. commitTransaction and abortTransaction are retryable write commands and MUST be retried according to the -[Retryable Writes Specification](../retryable-writes/retryable-writes.rst). +[Retryable Writes Specification](../retryable-writes/retryable-writes.md). Retryable writes and transactions both use the `txnNumber` associated with a ServerSession. For retryable writes, `txnNumber` would normally increment before each retryable command, whereas in a transaction, the `txnNumber` is @@ -860,7 +860,7 @@ execute a command directly with minimum additional client-side logic. This specification depends on: 1. [Driver Sessions Specification](../sessions/driver-sessions.rst) -2. [Retryable Writes Specification](../retryable-writes/retryable-writes.rst) +2. [Retryable Writes Specification](../retryable-writes/retryable-writes.md) ## **Backwards Compatibility** diff --git a/source/unified-test-format/unified-test-format.md b/source/unified-test-format/unified-test-format.md index 8b422d7094..3f5def1348 100644 --- a/source/unified-test-format/unified-test-format.md +++ b/source/unified-test-format/unified-test-format.md @@ -30,7 +30,7 @@ This test format can be used to define tests for the following specifications: - [CRUD](../crud/crud.md) - [GridFS](../gridfs/gridfs-spec.md) - [Retryable Reads](../retryable-reads/retryable-reads.md) -- [Retryable Writes](../retryable-writes/retryable-writes.rst) +- [Retryable Writes](../retryable-writes/retryable-writes.md) - [Sessions](../sessions/driver-sessions.rst) - [Transactions](../transactions/transactions.md) - [Convenient API for Transactions](../transactions-convenient-api/transactions-convenient-api.rst) diff --git a/source/uri-options/uri-options.rst b/source/uri-options/uri-options.rst index 67ed154445..8af993bc88 100644 --- a/source/uri-options/uri-options.rst +++ b/source/uri-options/uri-options.rst @@ -289,7 +289,7 @@ pertaining to URI options apply here. * - retryWrites - "true" or "false" - - defined in `retryable writes spec `_ + - defined in `retryable writes spec <../retryable-writes/retryable-writes.md#retrywrites>`_ - no - Enables retryable writes on server 3.6+