Skip to content

Commit

Permalink
Remove legacy test format from README
Browse files Browse the repository at this point in the history 
This also renumbers the prose tests
  • Loading branch information
@jmikola
jmikola committed Mar 6, 2024
1 parent 54fa6ed commit 5478c9d
Showing 1 changed file with 17 additions and 189 deletions.
206 changes: 17 additions & 189 deletions source/retryable-reads/tests/README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,192 +9,18 @@ Retryable Reads Tests
Introduction
============

The YAML and JSON files in the ``legacy`` and ``unified`` sub-directories are platform-independent tests
that drivers can use to prove their conformance to the Retryable Reads spec. Tests in the
``unified`` directory are written using the `Unified Test Format <../../unified-test-format/unified-test-format.md>`_.
Tests in the ``legacy`` directory are written using the format described below.
The YAML and JSON files in this directory are platform-independent tests meant
to exercise a driver's implementation of retryable reads. These tests utilize
the [Unified Test Format](../../unified-test-format/unified-test-format.md).

Prose tests, which are not easily expressed in YAML, are also presented
Several prose tests, which are not easily expressed in YAML, are also presented
in this file. Those tests will need to be manually implemented by each driver.

Tests will require a MongoClient created with options defined in the tests.
Integration tests will require a running MongoDB cluster with server versions
4.0 or later.
Prose Tests
===========

N.B. The spec specifies 3.6 as the minimum server version: however,
``failCommand`` is not supported on 3.6, so for now, testing requires MongoDB
4.0. Once `DRIVERS-560`_ is resolved, we will attempt to adapt its live failure
integration tests to test Retryable Reads on MongoDB 3.6.

.. _DRIVERS-560: https://jira.mongodb.org/browse/DRIVERS-560

Server Fail Point
=================

See: `Server Fail Point`_ in the Transactions spec test suite.

.. _Server Fail Point: ../../transactions/tests#server-fail-point

Disabling Fail Point after Test Execution
-----------------------------------------

After each test that configures a fail point, drivers should disable the
``failCommand`` fail point to avoid spurious failures in
subsequent tests. The fail point may be disabled like so::

db.runCommand({
configureFailPoint: "failCommand",
mode: "off"
});

Network Error Tests
===================

Network error tests are expressed in YAML and should be run against a standalone,
shard cluster, or single-node replica set.


Test Format
-----------

Each YAML file has the following keys:

- ``runOn`` (optional): An array of server version and/or topology requirements
for which the tests can be run. If the test environment satisfies one or more
of these requirements, the tests may be executed; otherwise, this file should
be skipped. If this field is omitted, the tests can be assumed to have no
particular requirements and should be executed. Each element will have some or
all of the following fields:

- ``minServerVersion`` (optional): The minimum server version (inclusive)
required to successfully run the tests. If this field is omitted, it should
be assumed that there is no lower bound on the required server version.

- ``maxServerVersion`` (optional): The maximum server version (inclusive)
against which the tests can be run successfully. If this field is omitted,
it should be assumed that there is no upper bound on the required server
version.

- ``topology`` (optional): An array of server topologies against which the
tests can be run successfully. Valid topologies are "single",
"replicaset", "sharded", and "load-balanced". If this field is omitted,
the default is all topologies (i.e. ``["single", "replicaset", "sharded",
"load-balanced"]``).

- ``serverless``: (optional): Whether or not the test should be run on Atlas
Serverless instances. Valid values are "require", "forbid", and "allow". If
"require", the test MUST only be run on Atlas Serverless instances. If
"forbid", the test MUST NOT be run on Atlas Serverless instances. If omitted
or "allow", this option has no effect.

The test runner MUST be informed whether or not Atlas Serverless is being
used in order to determine if this requirement is met (e.g. through an
environment variable or configuration option).

Note: the Atlas Serverless proxy imitates mongos, so the test runner is not
capable of determining if Atlas Serverless is in use by issuing commands
such as ``buildInfo`` or ``hello``. Furthermore, connections to Atlas
Serverless use a load balancer, so the topology will appear as
"load-balanced".

- ``database_name`` and ``collection_name``: Optional. The database and
collection to use for testing.

- ``bucket_name``: Optional. The GridFS bucket name to use for testing.

- ``data``: The data that should exist in the collection(s) under test before
each test run. This will typically be an array of documents to be inserted
into the collection under test (i.e. ``collection_name``); however, this field
may also be an object mapping collection names to arrays of documents to be
inserted into the specified collection.

- ``tests``: An array of tests that are to be run independently of each other.
Each test will have some or all of the following fields:

- ``description``: The name of the test.

- ``clientOptions``: Optional, parameters to pass to MongoClient().

- ``useMultipleMongoses`` (optional): If ``true``, and the topology type is
``Sharded``, the MongoClient for this test should be initialized with multiple
mongos seed addresses. If ``false`` or omitted, only a single mongos address
should be specified.

If ``true``, the topology type is ``LoadBalanced``, and Atlas Serverless is
not being used, the MongoClient for this test should be initialized with the
URI of the load balancer fronting multiple servers. If ``false`` or omitted,
the MongoClient for this test should be initialized with the URI of the load
balancer fronting a single server.

``useMultipleMongoses`` only affects ``Sharded`` and ``LoadBalanced``
topologies (excluding Atlas Serverless).

- ``skipReason``: Optional, string describing why this test should be skipped.

- ``failPoint``: Optional, a server fail point to enable, expressed as the
configureFailPoint command to run on the admin database.

- ``operations``: An array of documents describing an operation to be
executed. Each document has the following fields:

- ``name``: The name of the operation on ``object``.

- ``object``: The name of the object to perform the operation on. Can be
"database", "collection", "client", or "gridfsbucket."

- ``arguments``: Optional, the names and values of arguments.

- ``result``: Optional. The return value from the operation, if any. This
field may be a scalar (e.g. in the case of a count), a single document, or
an array of documents in the case of a multi-document read.

- ``error``: Optional. If ``true``, the test should expect an error or
exception.

- ``expectations``: Optional list of command-started events.

GridFS Tests
------------

GridFS tests are denoted by when the YAML file contains ``bucket_name``.
The ``data`` field will also be an object, which maps collection names
(e.g. ``fs.files``) to an array of documents that should be inserted into
the specified collection.

``fs.files`` and ``fs.chunks`` should be created in the database
specified by ``database_name``. This could be done via inserts or by
creating GridFSBuckets—using the GridFS ``bucketName`` (see
`GridFSBucket spec`_) specified by ``bucket_name`` field in the YAML
file—and calling ``upload_from_stream_with_id`` with the appropriate
data.

``Download`` tests should be tested against ``GridFS.download_to_stream``.
``DownloadByName`` tests should be tested against
``GridFS.download_to_stream_by_name``.


.. _GridFSBucket spec: https://github.com/mongodb/specifications/blob/master/source/gridfs/gridfs-spec.rst#configurable-gridfsbucket-class


Speeding Up Tests
-----------------

Drivers can greatly reduce the execution time of tests by setting `heartbeatFrequencyMS`_
and `minHeartbeatFrequencyMS`_ (internally) to a small value (e.g. 5ms), below what
is normally permitted in the SDAM spec. If a test specifies an explicit value for
heartbeatFrequencyMS (e.g. client or URI options), drivers MUST use that value.

.. _minHeartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#minheartbeatfrequencyms
.. _heartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms

Optional Enumeration Commands
=============================

A driver only needs to test the optional enumeration commands it has chosen to
implement (e.g. ``Database.listCollectionNames()``).

PoolClearedError Retryability Test
==================================
1. PoolClearedError Retryability Test
-------------------------------------

This test will be used to ensure drivers properly retry after encountering PoolClearedErrors.
It MUST be implemented by any driver that implements the CMAP specification.
Expand Down Expand Up @@ -232,8 +58,8 @@ This test requires MongoDB 4.2.9+ for ``blockConnection`` support in the failpoi

9. Disable the failpoint.

Retrying Reads in a Sharded Cluster
===================================
2. Retrying Reads in a Sharded Cluster
--------------------------------------

These tests will be used to ensure drivers properly retry reads on a different
mongos.
Expand All @@ -244,8 +70,8 @@ different mongos due to normal SDAM behavior of randomized suitable server
selection". Verify relevant code paths are correctly executed by the tests using
external means such as a logging, debugger, code coverage tool, etc.

Retryable Reads Are Retried on a Different mongos When One is Available
-----------------------------------------------------------------------
2.1 Retryable Reads Are Retried on a Different mongos When One is Available
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This test MUST be executed against a sharded cluster that has at least two
mongos instances, supports ``retryReads=true``, and has enabled the
Expand Down Expand Up @@ -278,8 +104,8 @@ mongos instances, supports ``retryReads=true``, and has enabled the
7. Disable the fail point on both ``s0`` and ``s1``.


Retryable Reads Are Retried on the Same mongos When No Others are Available
---------------------------------------------------------------------------
2.2 Retryable Reads Are Retried on the Same mongos When No Others are Available
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This test MUST be executed against a sharded cluster that supports
``retryReads=true`` and has enabled the ``configureFailPoint`` command
Expand Down Expand Up @@ -316,10 +142,12 @@ This test MUST be executed against a sharded cluster that supports
Changelog
=========

:2024-03-06: Convert legacy retryable reads tests to unified format.

:2024-02-21: Update mongos redirection prose tests to workaround SDAM behavior
preventing execution of deprioritization code paths.

:2023-08-26 Add prose tests for retrying in a sharded cluster.
:2023-08-26: Add prose tests for retrying in a sharded cluster.

:2022-04-22: Clarifications to ``serverless`` and ``useMultipleMongoses``.

Expand Down

0 comments on commit 5478c9d

Please sign in to comment.