DRIVERS-2789 Use Markdown for Specifications Documentation (#1482)

* DRIVERS-2789 Use Markdown for Specifications Documentation

* add changelog entry

* Update references

* Revert "Update references"

This reverts commit f30793e.

* Handle links to the markdown file

* formatting

* Update source/enumerate-collections.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/read-write-concern/read-write-concern.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/transactions/transactions.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/uri-options/uri-options.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/index-management/index-management.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/crud/crud.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update README.md

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* address review

* fix link

* Update source/change-streams/change-streams.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/uri-options/uri-options.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/uri-options/uri-options.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Update source/uri-options/uri-options.rst

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

* Wrap at 120 chars

* clean up csot changelog

---------

Co-authored-by: Jeremy Mikola <jmikola@gmail.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,7 +5,7 @@ Please complete the following before merging:
 - [ ] Update changelog.
 - [ ] Make sure there are generated JSON files from the YAML test files.
 - [ ] Test changes in at least one language driver.
-- [ ] Test these changes against all server versions and topologies (including standalone, replica set, sharded clusters, and serverless).
+- [ ] Test these changes against all server versions and topologies (including standalone, replica set, sharded
+  clusters, and serverless).
 
 <!-- See also: https://wiki.corp.mongodb.com/pages/viewpage.action?pageId=80806719 -->
-

.pre-commit-config.yaml

@@ -22,6 +22,19 @@ repos:
       args: ["--severity=error"]
       stages: [manual]
 
+- repo: https://github.com/executablebooks/mdformat
+  rev: 0.7.17
+  hooks:
+    - id: mdformat
+      args: ["--wrap=120"]
+      additional_dependencies:
+        [mdformat-gfm]
+
+- repo: https://github.com/tcort/markdown-link-check
+  rev: v3.11.2
+  hooks:
+    - id: markdown-link-check
+
 - repo: https://github.com/rstcheck/rstcheck
   rev: v6.2.0
   hooks:

README.md

@@ -0,0 +1,134 @@
+# MongoDB Specifications
+
+This repository holds in progress and completed specification for features of MongoDB, Drivers, and associated products.
+Also contained is a rudimentary system for producing these documents.
+
+## Driver Mantras
+
+When developing specifications -- and the drivers themselves -- we follow the following principles:
+
+### Strive to be idiomatic, but favor consistency
+
+Drivers attempt to provide the easiest way to work with MongoDB in a given language ecosystem, while specifications
+attempt to provide a consistent behavior and experience across all languages. Drivers should strive to be as idiomatic
+as possible while meeting the specification and staying true to the original intent.
+
+### No Knobs
+
+Too many choices stress out users. Whenever possible, we aim to minimize the number of configuration options exposed to
+users. In particular, if a typical user would have no idea how to choose a correct value, we pick a good default instead
+of adding a knob.
+
+### Topology agnostic
+
+Users test and deploy against different topologies or might scale up from replica sets to sharded clusters. Applications
+should never need to use the driver differently based on topology type.
+
+### Where possible, depend on server to return errors
+
+The features available to users depend on a server's version, topology, storage engine and configuration. So that
+drivers don't need to code and test all possible variations, and to maximize forward compatibility, always let users
+attempt operations and let the server error when it can't comply. Exceptions should be rare: for cases where the server
+might not error and correctness is at stake.
+
+### Minimize administrative helpers
+
+Administrative helpers are methods for admin tasks, like user creation. These are rarely used and have maintenance costs
+as the server changes the administrative API. Don't create administrative helpers; let users rely on "RunCommand" for
+administrative commands.
+
+### Check wire version, not server version
+
+When determining server capabilities within the driver, rely only on the maxWireVersion in the hello response, not on
+the X.Y.Z server version. An exception is testing server development releases, as the server bumps wire version early
+and then continues to add features until the GA.
+
+### When in doubt, use "MUST" not "SHOULD" in specs
+
+Specs guide our work. While there are occasionally valid technical reasons for drivers to differ in their behavior,
+avoid encouraging it with a wishy-washy "SHOULD" instead of a more assertive "MUST".
+
+### Defy augury
+
+While we have some idea of what the server will do in the future, don't design features with those expectations in mind.
+Design and implement based on what is expected in the next release.
+
+Case Study: In designing OP_MSG, we held off on designing support for Document Sequences in Replies in drivers until the
+server would support it. We subsequently decided not to implement that feature in the server.
+
+### The best way to see what the server does is to test it
+
+For any unusual case, relying on documentation or anecdote to anticipate the server's behavior in different
+versions/topologies/etc. is error-prone. The best way to check the server's behavior is to use a driver or the shell and
+test it directly.
+
+### Drivers follow semantic versioning
+
+Drivers should follow X.Y.Z versioning, where breaking API changes require a bump to X. See
+[semver.org](https://semver.org/) for more.
+
+### Backward breaking behavior changes and semver
+
+Backward breaking behavior changes can be more dangerous and disruptive than backward breaking API changes. When
+thinking about the implications of a behavior change, ask yourself what could happen if a user upgraded your library
+without carefully reading the changelog and/or adequately testing the change.
+
+## Writing Documents
+
+Write documents using [GitHub Flavored Markdown](https://github.github.com/gfm/), following the
+[MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/).
+
+Store all source documents in the `source/` directory.
+
+## Linting
+
+This repo uses [pre-commit](https://pypi.org/project/pre-commit/) for managing linting. `pre-commit` performs various
+checks on the files and uses tools that help follow a consistent style within the repo.
+
+To set up `pre-commit` locally, run:
+
+```bash
+brew install pre-commit
+pre-commit install
+```
+
+To run `pre-commit` manually, run `pre-commit run --all-files`.
+
+To run a manual hook like `shellcheck` manually, run:
+
+```bash
+pre-commit run --all-files --hook-stage manual shellcheck
+```
+
+## Prose test numbering
+
+When numbering prose tests, always use relative numbered bullets (`1.`). New tests must be appended at the end of the
+test list, since drivers may refer to existing tests by number.
+
+Outdated tests must not be removed completely, but may be marked as such (e.g. by striking through or replacing the
+entire test with a note (e.g. *Removed*).
+
+## Building Documents
+
+We build the docs in `text` mode in CI to make sure they build without errors. We don't actually support building html,
+since we rely on GitHub to render the documents. To build locally, run:
+
+```bash
+pip install sphinx
+cd source 
+sphinx-build -W -b text . docs_build index.rst
+```
+
+## Converting to JSON
+
+There are many YAML to JSON converters. There are even several converters called `yaml2json` in NPM. Alas, we are not
+using `yaml2json` anymore, but instead the [js-yaml](https://www.npmjs.com/package/js-yaml) package. Use only that
+converter, so that JSON is formatted consistently.
+
+Run `npm install -g js-yaml`, then run `make` in the `source` directory at the top level of this repository to convert
+all YAML test files to JSON.
+
+## Licensing
+
+All the specs in this repository are available under the
+[Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License](https://creativecommons.org/licenses/by-nc-sa/3.0/us/).

scripts/README.md

@@ -0,0 +1,38 @@
+# Scripts for use with Drivers specifications repository
+
+## migrate_to_md
+
+Use this file to automate the process of converting a specification from rst to GitHub Flavored Markdown.
+
+The goal of the script is to do most of the work, including updating relative links to the new document from other
+files. Note that all known features will translate, including explicit cross-reference markers (e.g. `.. _foo`) which
+are translated to a `<div id="foo">`. It will also create a new changelog entry with today's date marking the
+conversion.
+
+### Prerequisites
+
+```bash
+brew install pandoc
+brew install pre-commit
+brew install python # or get python through your preferred channel
+pre-commit install
+```
+
+### Usage
+
+- Run the script as:
+
+```bash
+python3 scripts/migrate_to_md.py "source/<path_to_rst_file>"
+```
+
+- Address any errors that were printed during the run.
+
+- Ensure that the generated markdown file is properly formatted.
+
+- Ensure that the links in the new file are working, by running `pre-commit run markdown-link-check` and addressing
+  failures until that passes.
+
+- Remove the rst file using `git rm`.
+
+- Create a PR. When you commit the changes, the `mdformat` `pre-commit` hook will update the formatting as appropriate.