From e29f381cb2022ffe6c5790a6ddff02b6bf0a5b7f Mon Sep 17 00:00:00 2001 From: Shane Harvey Date: Mon, 14 Aug 2023 13:44:50 -0700 Subject: [PATCH] DRIVERS-2578 Drivers use polling SDAM on AWS Lambda Disable streaming SDAM by default on AWS Lambda and similar FaaS platforms. Introduce the sdamMode=stream/poll/auto URI option. --- .../faas-automated-testing.rst | 5 + .../server-monitoring.rst | 92 +++++++++++-- .../tests/unified/sdamMode.json | 129 ++++++++++++++++++ .../tests/unified/sdamMode.yml | 67 +++++++++ source/uri-options/tests/sdam-options.json | 46 +++++++ source/uri-options/tests/sdam-options.yml | 35 +++++ source/uri-options/uri-options.rst | 11 +- 7 files changed, 374 insertions(+), 11 deletions(-) create mode 100644 source/server-discovery-and-monitoring/tests/unified/sdamMode.json create mode 100644 source/server-discovery-and-monitoring/tests/unified/sdamMode.yml create mode 100644 source/uri-options/tests/sdam-options.json create mode 100644 source/uri-options/tests/sdam-options.yml diff --git a/source/faas-automated-testing/faas-automated-testing.rst b/source/faas-automated-testing/faas-automated-testing.rst index df5c3a7408..ef9be1a0b2 100644 --- a/source/faas-automated-testing/faas-automated-testing.rst +++ b/source/faas-automated-testing/faas-automated-testing.rst @@ -243,6 +243,8 @@ the function implementation the driver MUST: - Drivers MUST record the durations and counts of the heartbeats, the durations of the commands, as well as keep track of the number of open connections, and report this information in the function response as JSON. +- Drivers MUST assert no ServerHeartbeat events contain the ``awaited=True`` flag to + confirm that the streaming protocol is disabled (`DRIVERS-2578`_). Running in Continuous Integration @@ -368,6 +370,9 @@ Description of the behaviour of run-deployed-lambda-aws-tests.sh: Changelog ========= +:2023-08-21: Drivers MUST assert that the streaming protocol is disabled in the Lambda function. :2023-08-17: Fixed URI typo, added host note, increase assume role duration. :2023-06-22: Updated evergreen configuration to use task groups. :2023-04-14: Added list of supported variants, added additional template config. + +.. _DRIVERS-2578: https://jira.mongodb.org/browse/DRIVERS-2578 diff --git a/source/server-discovery-and-monitoring/server-monitoring.rst b/source/server-discovery-and-monitoring/server-monitoring.rst index 940366a9a7..2a94bdbcb5 100644 --- a/source/server-discovery-and-monitoring/server-monitoring.rst +++ b/source/server-discovery-and-monitoring/server-monitoring.rst @@ -91,6 +91,28 @@ Round trip time. The client's measurement of the duration of one hello or legacy The RTT is used to support `localThresholdMS`_ from the Server Selection spec and `timeoutMS`_ from the `Client Side Operations Timeout Spec`_. +FaaS +```` + +A Function-as-a-Service (FaaS) environment like AWS Lambda. + +sdamMode +```````` + +The sdamMode option configures which server monitoring protocol to use. Valid modes are +"stream", "poll", or "auto". The default value MUST be "auto": + +- With "stream" mode, the client MUST use the streaming protocol when the server supports + it or fall back to the polling protocol otherwise. +- With "poll" mode, the client MUST use the polling protocol. +- With "auto" mode, the client MUST behave the same as "poll" mode when running on a FaaS + platform or the same as "stream" mode otherwise. The client detects that it's + running on a FaaS platform via the same rules for generating the ``client.env`` + handshake metadata field in the `MongoDB Handshake spec`_. + +Multi-threaded or asynchronous drivers MUST implement this option. +See `Why disable the streaming protocol on FaaS platforms like AWS Lambda?`_ and +`Why introduce a knob for sdamMode?`_ Monitoring '''''''''' @@ -203,7 +225,7 @@ Clients use the streaming protocol when supported ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When a monitor discovers that the server supports the streamable hello or legacy hello -command, it MUST use the `streaming protocol`_. +command and the client does not have `streaming disabled`_, it MUST use the `streaming protocol`_. Single-threaded monitoring `````````````````````````` @@ -491,6 +513,22 @@ connected to a server that supports the awaitable hello or legacy hello commands This protocol requires an extra thread and an extra socket for each monitor to perform RTT calculations. +.. _streaming is disabled: + +Streaming disabled +`````````````````` + +The streaming protocol MUST be disabled when either: + +- the client is configured with sdamMode=poll, or +- the client is configured with sdamMode=auto and a FaaS platform is detected, or +- the server does not support streaming (eg MongoDB <4.4). + +When the streaming protocol is disabled the client MUST use the `polling protocol`_ +and MUST NOT start an extra thread or connection for `Measuring RTT`_. + +See `Why disable the streaming protocol on FaaS platforms like AWS Lambda?`_. + Streaming hello or legacy hello ``````````````````````````````` @@ -584,8 +622,8 @@ current monitoring connection. (See `Drivers cancel in-progress monitor checks`_ Polling Protocol '''''''''''''''' -The polling protocol is used to monitor MongoDB <= 4.4 servers. The client -`checks`_ a server with a hello or legacy hello command and then sleeps for +The polling protocol is used to monitor MongoDB <= 4.4 servers or when `streaming is disabled`_. +The client `checks`_ a server with a hello or legacy hello command and then sleeps for heartbeatFrequencyMS before running another check. Marking the connection pool as ready (CMAP only) @@ -661,6 +699,12 @@ The event API here is assumed to be like the standard `Python Event heartbeatFrequencyMS = heartbeatFrequencyMS minHeartbeatFrequencyMS = 500 stableApi = stableApi + if sdamMode == "stream": + streamingEnabled = True + elif sdamMode == "poll": + streamingEnabled = False + else: # sdamMode == "auto" + streamingEnabled = not isFaas() # Internal Monitor state: connection = Null @@ -671,8 +715,6 @@ The event API here is assumed to be like the standard `Python Event rttMonitor = RttMonitor(serverAddress, stableApi) def run(): - # Start the RttMonitor. - rttMonitor.run() while this monitor is not stopped: previousDescription = description try: @@ -700,7 +742,10 @@ The event API here is assumed to be like the standard `Python Event serverSupportsStreaming = description.type != Unknown and description.topologyVersion != Null connectionIsStreaming = connection != Null and connection.moreToCome transitionedWithNetworkError = isNetworkError(description.error) and previousDescription.type != Unknown - if serverSupportsStreaming or connectionIsStreaming or transitionedWithNetworkError: + if streamingEnabled and serverSupportsStreaming and not rttMonitor.started: + # Start the RttMonitor. + rttMonitor.run() + if (streamingEnabled and (serverSupportsStreaming or connectionIsStreaming)) or transitionedWithNetworkError: continue wait() @@ -733,13 +778,13 @@ The event API here is assumed to be like the standard `Python Event response = connection.handshakeResponse elif connection.moreToCome: response = read next helloCommand exhaust response - elif previousDescription.topologyVersion: + elif streamingEnabled and previousDescription.topologyVersion: # Initiate streaming hello or legacy hello if connectTimeoutMS != 0: set connection timeout to connectTimeoutMS+heartbeatFrequencyMS response = call {helloCommand: 1, helloOk: True, topologyVersion: previousDescription.topologyVersion, maxAwaitTimeMS: heartbeatFrequencyMS} else: - # The server does not support topologyVersion. + # The server does not support topologyVersion or streamingEnabled=False. response = call {helloCommand: 1, helloOk: True} # If the server supports hello, then response.helloOk will be true @@ -1140,6 +1185,32 @@ the "awaited" field on server heartbeat events so that applications can differentiate a slow heartbeat in the polling protocol from a normal awaitable hello or legacy hello heartbeat in the new protocol. +Why disable the streaming protocol on FaaS platforms like AWS Lambda? +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +The streaming protocol requires an extra connection and thread per monitored +server which is expensive on platforms like AWS Lambda. The extra connection +is particularly inefficient when thousands of AWS instances and thus +thousands of clients are used. + +Additionally, the streaming protocol relies on the assumption that the client +can read the server's heartbeat responses in a timely manner, otherwise the +client will be acting on stale information. In many FaaS platforms, like AWS +Lambda, host applications will be suspended and resumed many minutes later. +This behavior causes a build up of heartbeat responses and the client can end +up spending a long time in a catch up phase processing outdated responses. +This problem was discovered in `DRIVERS-2246`_. + +We decided to make polling the default behavior when running on FaaS platforms +like AWS Lambda to improve scalability, performance, and reliability. + +Why introduce a knob for sdamMode? +'''''''''''''''''''''''''''''''''' + +The sdamMode knob provides an workaround in cases where the polling +protocol would be a better choice but the driver is not running on a FaaS +platform. It also provides a workaround in case the FaaS detection +logic becomes outdated or inaccurate. Changelog --------- @@ -1159,6 +1230,7 @@ Changelog :2022-04-05: Preemptively cancel in progress operations when SDAM heartbeats timeout. :2022-10-05: Remove spec front matter reformat changelog. :2022-11-17: Add minimum RTT tracking and remove 90th percentile RTT. +:2023-08-21: Add sdamMode and default to the Polling Protocol on FaaS. ---- @@ -1183,4 +1255,6 @@ Changelog .. _Why synchronize clearing a server's pool with updating the topology?: server-discovery-and-monitoring.rst#why-synchronize-clearing-a-server-s-pool-with-updating-the-topology? .. _Client Side Operations Timeout Spec: /source/client-side-operations-timeout/client-side-operations-timeout.rst .. _timeoutMS: /source/client-side-operations-timeout/client-side-operations-timeout.rst#timeoutMS -.. _Why does the pool need to support closing in use connections as part of its clear logic?: /source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#Why-does-the-pool-need-to-support-closing-in-use-connections-as-part-of-its-clear-logic? \ No newline at end of file +.. _Why does the pool need to support closing in use connections as part of its clear logic?: /source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#Why-does-the-pool-need-to-support-closing-in-use-connections-as-part-of-its-clear-logic? +.. _DRIVERS-2246: https://jira.mongodb.org/browse/DRIVERS-2246 +.. _MongoDB Handshake spec: /source/mongodb-handshake/handshake.rst#client-env diff --git a/source/server-discovery-and-monitoring/tests/unified/sdamMode.json b/source/server-discovery-and-monitoring/tests/unified/sdamMode.json new file mode 100644 index 0000000000..0df841788d --- /dev/null +++ b/source/server-discovery-and-monitoring/tests/unified/sdamMode.json @@ -0,0 +1,129 @@ +{ + "description": "sdamMode", + "schemaVersion": "1.3", + "tests": [ + { + "description": "connect with sdamMode=auto", + "operations": [ + { + "name": "createEntities", + "object": "testRunner", + "arguments": { + "entities": [ + { + "client": { + "id": "client0", + "uriOptions": { + "sdamMode": "auto" + } + } + }, + { + "database": { + "id": "dbSdamModeAuto", + "client": "client0", + "databaseName": "sdam-tests" + } + } + ] + } + }, + { + "name": "runCommand", + "object": "dbSdamModeAuto", + "arguments": { + "commandName": "ping", + "command": { + "ping": 1 + } + }, + "expectResult": { + "ok": 1 + } + } + ] + }, + { + "description": "connect with sdamMode=stream", + "operations": [ + { + "name": "createEntities", + "object": "testRunner", + "arguments": { + "entities": [ + { + "client": { + "id": "client1", + "uriOptions": { + "sdamMode": "stream" + } + } + }, + { + "database": { + "id": "dbSdamModeStream", + "client": "client1", + "databaseName": "sdam-tests" + } + } + ] + } + }, + { + "name": "runCommand", + "object": "dbSdamModeStream", + "arguments": { + "commandName": "ping", + "command": { + "ping": 1 + } + }, + "expectResult": { + "ok": 1 + } + } + ] + }, + { + "description": "connect with sdamMode=poll", + "operations": [ + { + "name": "createEntities", + "object": "testRunner", + "arguments": { + "entities": [ + { + "client": { + "id": "client2", + "uriOptions": { + "sdamMode": "poll" + } + } + }, + { + "database": { + "id": "dbSdamModePoll", + "client": "client2", + "databaseName": "sdam-tests" + } + } + ] + } + }, + { + "name": "runCommand", + "object": "dbSdamModePoll", + "arguments": { + "commandName": "ping", + "command": { + "ping": 1 + } + }, + "expectResult": { + "ok": 1 + } + } + ] + } + ] +} diff --git a/source/server-discovery-and-monitoring/tests/unified/sdamMode.yml b/source/server-discovery-and-monitoring/tests/unified/sdamMode.yml new file mode 100644 index 0000000000..84d2a33648 --- /dev/null +++ b/source/server-discovery-and-monitoring/tests/unified/sdamMode.yml @@ -0,0 +1,67 @@ +description: sdamMode + +schemaVersion: "1.3" + +tests: + - description: "connect with sdamMode=auto" + operations: + - name: createEntities + object: testRunner + arguments: + entities: + - client: + id: &client0 client0 + uriOptions: + sdamMode: "auto" + - database: + id: &dbSdamModeAuto dbSdamModeAuto + client: *client0 + databaseName: sdam-tests + - name: runCommand + object: *dbSdamModeAuto + arguments: + commandName: ping + command: { ping: 1 } + expectResult: { ok: 1 } + + - description: "connect with sdamMode=stream" + operations: + - name: createEntities + object: testRunner + arguments: + entities: + - client: + id: &client1 client1 + uriOptions: + sdamMode: "stream" + - database: + id: &dbSdamModeStream dbSdamModeStream + client: *client1 + databaseName: sdam-tests + - name: runCommand + object: *dbSdamModeStream + arguments: + commandName: ping + command: { ping: 1 } + expectResult: { ok: 1 } + + - description: "connect with sdamMode=poll" + operations: + - name: createEntities + object: testRunner + arguments: + entities: + - client: + id: &client2 client2 + uriOptions: + sdamMode: "poll" + - database: + id: &dbSdamModePoll dbSdamModePoll + client: *client2 + databaseName: sdam-tests + - name: runCommand + object: *dbSdamModePoll + arguments: + commandName: ping + command: { ping: 1 } + expectResult: { ok: 1 } diff --git a/source/uri-options/tests/sdam-options.json b/source/uri-options/tests/sdam-options.json new file mode 100644 index 0000000000..afc39ca3f5 --- /dev/null +++ b/source/uri-options/tests/sdam-options.json @@ -0,0 +1,46 @@ +{ + "tests": [ + { + "description": "sdamMode=auto", + "uri": "mongodb://example.com/?sdamMode=auto", + "valid": true, + "warning": false, + "hosts": null, + "auth": null, + "options": { + "sdamMode": "auto" + } + }, + { + "description": "sdamMode=stream", + "uri": "mongodb://example.com/?sdamMode=stream", + "valid": true, + "warning": false, + "hosts": null, + "auth": null, + "options": { + "sdamMode": "stream" + } + }, + { + "description": "sdamMode=poll", + "uri": "mongodb://example.com/?sdamMode=poll", + "valid": true, + "warning": false, + "hosts": null, + "auth": null, + "options": { + "sdamMode": "poll" + } + }, + { + "description": "invalid sdamMode", + "uri": "mongodb://example.com/?sdamMode=invalid", + "valid": true, + "warning": true, + "hosts": null, + "auth": null, + "options": {} + } + ] +} diff --git a/source/uri-options/tests/sdam-options.yml b/source/uri-options/tests/sdam-options.yml new file mode 100644 index 0000000000..735bf0aed7 --- /dev/null +++ b/source/uri-options/tests/sdam-options.yml @@ -0,0 +1,35 @@ +tests: + - description: "sdamMode=auto" + uri: "mongodb://example.com/?sdamMode=auto" + valid: true + warning: false + hosts: ~ + auth: ~ + options: + sdamMode: "auto" + + - description: "sdamMode=stream" + uri: "mongodb://example.com/?sdamMode=stream" + valid: true + warning: false + hosts: ~ + auth: ~ + options: + sdamMode: "stream" + + - description: "sdamMode=poll" + uri: "mongodb://example.com/?sdamMode=poll" + valid: true + warning: false + hosts: ~ + auth: ~ + options: + sdamMode: "poll" + + - description: "invalid sdamMode" + uri: "mongodb://example.com/?sdamMode=invalid" + valid: true + warning: true + hosts: ~ + auth: ~ + options: {} diff --git a/source/uri-options/uri-options.rst b/source/uri-options/uri-options.rst index 035e7ddec5..3f4589b4a5 100644 --- a/source/uri-options/uri-options.rst +++ b/source/uri-options/uri-options.rst @@ -282,17 +282,23 @@ pertaining to URI options apply here. - The name of the replica set to connect to * - retryReads - - "true" or "false + - "true" or "false" - defined in `retryable reads spec `_ - no - Enables retryable reads on server 3.6+ * - retryWrites - - "true" or "false + - "true" or "false" - defined in `retryable writes spec `_ - no - Enables retryable writes on server 3.6+ + * - sdamMode + - "stream", "poll", or "auto" + - defined in `SDAM spec `__ + - required for multi-threaded or asynchronous drivers + - Configures which server monitoring protocol to use. + * - serverSelectionTimeoutMS - positive integer; a driver may also accept 0 to be used for a special case, provided that it documents the meaning - defined in `server selection spec `__ @@ -525,6 +531,7 @@ this specification MUST be updated to reflect those changes. Changelog --------- +:2023-08-21: Add sdamMode option. :2022-10-05: Remove spec front matter and reformat changelog. :2022-01-19: Add the timeoutMS option and deprecate some existing timeout options :2021-12-14: Add SOCKS5 options