🔗 Enable external MyST project cross-references (#1180)

* 🚨 Warn on duplicate identifiers across project pages
* 🔧 Write myst.xref.json file to site
* 🔧 Improve frontmatter reference validation
* 🦋 Change loadIntersphinx -> loadReferences and include MyST xrefs
* 🔧 Add kind and /content to myst.xref.json
* 🔧 Old MystTransformer -> SphinxTransformer with some additional deprecation warnings
* 🔗 Add link transform for myst xrefs
* 🔗 Split out function for getting link text from target node
* 🔗 Add transform to add link text from myst xrefs
* 🔗 Add identifiers for page slugs to myst xrefs
* 🏷  Alias name/label in project/page frontmatter and respect as identifiers
* 🔗 Add check for empty link text
* 🔧 Remove reserved reference keys, add regex
* 🔧 Only allow label in page frontmatter, not project
* 🔗 Update xref link transformer for <xref:id/path#target> format
* 📚 Update docs for myst xrefs
* 🔗 Fill enumerator strings for external project xrefs
* 🐛 Ignore unwanted identifier nodes in myst xref transform

---------

Co-authored-by: Rowan Cockett <rowanc1@gmail.com>
Co-authored-by: Chris Holdgraf <choldgraf@berkeley.edu>

.changeset/beige-turkeys-divide.md

@@ -0,0 +1,6 @@
+---
+'myst-transforms': patch
+'myst-cli': patch
+---
+
+Change loadIntersphinx -> loadReferences and include MyST xrefs

.changeset/chilled-trains-drive.md

@@ -0,0 +1,6 @@
+---
+'myst-frontmatter': patch
+'myst-cli': patch
+---
+
+Alias name/label in prjoect/page frontmatter and respect as identifiers

.changeset/eleven-coats-obey.md

@@ -0,0 +1,5 @@
+---
+'myst-transforms': patch
+---
+
+Split out function for getting link text from target node

.changeset/fast-kiwis-allow.md

@@ -0,0 +1,5 @@
+---
+'myst-cli': patch
+---
+
+Write myst.xref.json file to site

.changeset/hungry-hats-shop.md

@@ -0,0 +1,5 @@
+---
+"myst-spec-ext": patch
+---
+
+Add CrossReference

.changeset/hungry-pandas-protect.md

@@ -0,0 +1,5 @@
+---
+'myst-cli': patch
+---
+
+Fill enumerator strings for external project xrefs

.changeset/neat-sheep-grow.md

@@ -0,0 +1,5 @@
+---
+'myst-cli': patch
+---
+
+Add transform to add link text from myst xrefs

.changeset/rare-wasps-do.md

@@ -0,0 +1,6 @@
+---
+'myst-transforms': patch
+'myst-cli': patch
+---
+
+Warn on duplicate identifiers across project pages

.changeset/sharp-knives-dream.md

@@ -0,0 +1,7 @@
+---
+'myst-transforms': patch
+'myst-common': patch
+'myst-cli': patch
+---
+
+Add check for empty link text

.changeset/small-fireants-cry.md

@@ -0,0 +1,5 @@
+---
+"myst-cli": patch
+---
+
+Do not build pages that do not exist for static html export

.changeset/weak-coats-sing.md

@@ -0,0 +1,5 @@
+---
+'myst-frontmatter': patch
+---
+
+Improve frontmatter reference validation

.changeset/wet-apricots-tan.md

@@ -0,0 +1,5 @@
+---
+'myst-transforms': patch
+---
+
+Add link transform for myst xrefs

docs/_toc.yml

@@ -73,6 +73,7 @@ parts:
       - file: document-parts
       - file: settings
       - file: glossary
+      - file: xref
   - caption: Contribute
     chapters:
       - file: contributing

docs/background.md

@@ -9,7 +9,7 @@ This page discusses high-level questions about the MyST Markdown ecosystem, hist
 
 MyST Markdown (Markedly Structured Text) is a markup language that builds on standard markdown and is designed to create publication-quality documents, books, presentations, and websites written entirely in Markdown. The [ExecutableBooks] team received a grant from the [Sloan Foundation](https://sloan.org) to build, enhance, and promote a new path to document creation and publishing for next-generation scientific textbooks and lectures ([Grant #9231](https://sloan.org/grant-detail/9231)).
 
-The initial use case driving the development and design of MyST Markdown has been [JupyterBook], which allows you to create educational online textbooks and tutorials with Jupyter Notebooks and narrative content written in MyST. The extensions and design of MyST is inspired by the [Sphinx] and [reStructuredText](https://docutils.sourceforge.io/rst.html) (RST) ecosystems. Jupyter Book is considered a [distribution of Sphinx](myst:jupyterbook#explain/sphinx), and builds on the Sphinx and [Docutils] Python packages.
+The initial use case driving the development and design of MyST Markdown has been [JupyterBook], which allows you to create educational online textbooks and tutorials with Jupyter Notebooks and narrative content written in MyST. The extensions and design of MyST is inspired by the [Sphinx] and [reStructuredText](https://docutils.sourceforge.io/rst.html) (RST) ecosystems. Jupyter Book is considered a [distribution of Sphinx](xref:jupyterbook#explain/sphinx), and builds on the Sphinx and [Docutils] Python packages.
 
 MyST Markdown enables rich content generation and is a powerful textual format for scientific and technical communication with potential for broad adoption in modern publishing workflows. In 2022, the Executable Books team started work to document the specification behind the markup language, called [myst-spec](https://github.com/executablebooks/myst-spec), this work has enabled other tools and implementations in the scientific ecosystem to build on MyST Markdown (e.g. [scientific authoring tools](https://curvenote.com/for/writing), and [documentation systems](https://blog.readthedocs.com/jupyter-book-read-the-docs/)).
 

docs/creating-pdf-documents.md

@@ -54,7 +54,7 @@ The default PDF renderer uses $\LaTeX$ to create PDFs, which means that to work
 As an alternative, for faster PDF builds, you may use [Typst](#rendering-pdfs-with-typst) instead.
 ```
 
-The rendering process for scientific PDFs uses $\LaTeX$ and makes use of the [`jtex`](myst:jtex) templating library, to convert to $\LaTeX$ the `myst-to-tex` packages is used. The libraries work together for sharing information about [frontmatter](./frontmatter.md) (e.g. title, keywords, authors, and affiliations).
+The rendering process for scientific PDFs uses $\LaTeX$ and makes use of the [`jtex`](xref:jtex) templating library, to convert to $\LaTeX$ the `myst-to-tex` packages is used. The libraries work together for sharing information about [frontmatter](./frontmatter.md) (e.g. title, keywords, authors, and affiliations).
 
 ```{mermaid}
 flowchart LR
@@ -95,7 +95,7 @@ Ensure that you download a full distribution with appropriate libraries installe
 
 MyST also provides an option to build PDFs with [Typst](https://typst.app). Typst is a markup-based typesetting language. Compared to $\LaTeX$, syntax is streamlined and consistent, and compile time is significantly faster. To render Typst PDFs locally, you must install the [Typst CLI](https://github.com/typst/typst).
 
-To add Typst to your export targets, add `format: typst` and select a Typst template. These templates use the same [MyST templating library](myst:jtex) as $\LaTeX$ templates to support document [frontmatter](./frontmatter.md).
+To add Typst to your export targets, add `format: typst` and select a Typst template. These templates use the same [MyST templating library](xref:jtex) as $\LaTeX$ templates to support document [frontmatter](./frontmatter.md).
 
 ```{code-block} yaml
 :filename: article.md
@@ -167,7 +167,7 @@ abstract: |
 
 ### Template `options`
 
-Template authors should choose to use [standard frontmatter](./frontmatter.md) properties where possible, however, all templates can also expose custom options through their [](/jtex/template-yml). Include options for the build in the exports list. For example, to turn on `line_numbers` in the template, add the option to the dictionary.
+Template authors should choose to use [standard frontmatter](./frontmatter.md) properties where possible, however, all templates can also expose custom options through their [](xref:jtex#template-yml). Include options for the build in the exports list. For example, to turn on `line_numbers` in the template, add the option to the dictionary.
 
 ```{code-block} yaml
 :linenos:

docs/cross-references.md

@@ -20,7 +20,7 @@ See [](./citations.md) to cite scholarly work and create bibliographies.
 
 ## Directive Targets
 
-Targets are custom anchors that you can refer to elsewhere, for example, a figure, section, table, program, or proof. To be referenced, they must have a `label`/`identifier` pair [in the AST](myst:spec#association). These can be created by setting the `label` option in many directives. For example, to label and reference a figure, use the following syntax:
+Targets are custom anchors that you can refer to elsewhere, for example, a figure, section, table, program, or proof. To be referenced, they must have a `label`/`identifier` pair [in the AST](xref:spec#association). These can be created by setting the `label` option in many directives. For example, to label and reference a figure, use the following syntax:
 
 % TODO: fix equation label redundancy here would nice to be able to simplify the onboarding (just use label, same as tex and ast)
 
@@ -326,4 +326,4 @@ Finally, under the `numbering` object, you may specify `enumerator`. For now, th
 ```yaml
 numbering:
   enumerator: A1.%s
-```
+```

docs/documents-exports.md

@@ -27,7 +27,7 @@ Below are supported export types and links to documentation for further reading:
 ```
 
 ```{seealso} The MyST templating engine drives document exports
-You can also explore the [MyST templating](myst:jtex) documentation for a deeper dive into defining templates.
+You can also explore the [MyST templating](xref:jtex) documentation for a deeper dive into defining templates.
 ```
 
 ## Building Exports

docs/external-references.md

@@ -18,27 +18,87 @@ MyST supports referencing rich content in a growing number of formats, including
 [](./cross-references.md) for referencing content in your project and [](./citations.md) to cite scholarly work and create bibliographies.
 ```
 
+(myst-xref)=
+
 ## Referencing external MyST projects
 
 When using the HTML renderer for MyST, an API is provided for the deployed site.
-This provides pre-parsed, structured content as an AST that can be included and rendered in a tooltip.
+This provides pre-parsed, structured content as an AST that can be included in other projects and rendered in a tooltip.
 
-```{tip}
-Try adding `.json` at the end of the URL on this page. The data is structured and provides authors, license information,
+:::{tip} Add `.json` to any MyST URL to access the structured data
+All MyST pages come with a structured data representation that provides authors, license information,
 as well as the full content in a parsed form that can be used for an inline reference on external pages.
+Try adding `.json` at the end of the URL on this page.
+:::
+
+In your project configuration, include the `references` object with named links out to the external MyST projects that you will reference in your project. The example below shows how you would load cross-references that are pulled from the MyST Specification documentation.
+
+(myst-xref-config)=
+
+```yaml
+references:
+  spec: https://mystmd.org/spec
 ```
 
-```{note}
-Currently cross-project links aren't fully implemented, check back soon!
+When you specify these in your project configuration, MyST will load the cross-references and provide access to all of the pages and reference targets in that project.
+References are cached to disk locally in the `_build` folder, eliminating duplicate web requests on subsequent builds.
+To delete the cache and manually re-load the references, run `myst clean --cache`.
+
+````{important}
+# MyST Cross Reference Examples
+
+```{list-table}
+:header-rows: 1
+* - MyST Syntax
+  - Rendered
+* - `[](xref:spec)`
+    : A reference to the first page of the MyST Spec.
+  - [](xref:spec)
+* - `[](xref:spec#paragraph)`
+    : A reference to the node "paragraph" target in the MyST Spec.
+  - [](xref:spec#paragraph)
+* - `<xref:spec/tables#example>`
+    : Alternate link syntax that references a heading on a specific page of the spec documentation.
+  - <xref:spec/tables#example>
 ```
+````
+
+To reference content in the linked MyST project, use the `xref:` protocol in a link followed by the `project` key (from `references`), the url path and the target.
+For example, `<xref:spec/tables#example>` renders to:\
+"<xref:spec/tables#example>"\
+and is made of a `protocol`, `project`, `path` and `target`.
+
+protocol
+: The protocol for this type of link is `xref:`, and is what selects for cross-project referencing.
+
+project
+: the `project` key above is "spec" which is defined in your local [project configuration](#myst-xref-config) above.
+
+path
+: the `path` is everything that follows the project before the `#`. It corresponds directly to the URL path in the MyST site. In the example above, path is `/tables`.
+: Path is **optional** in most cases; the `target` can be resolved against the entire project without the path. An exception to this is headings without an explicit label in the source markdown — these will require path. Note that these references may not be intended to be persistent by the source author, so use caution in linking to them!
+
+target
+: The target is everything that follows the `#` and is a named reference in the project. In the example above it is "syntax".
+: Target is also optional - if not provided, the cross reference will simply link to a page in the external MyST project. However, without the target, there will not be a rendered tooltip.
+: When creating a cross reference, you can determine `path` and `target` by simply navigating to your target content in the external MyST project and copying the path and fragment from the URL.
+
+If no link text is provided, e.g. `[](xref:...)`, text will be generated from the external project at build-time. You may override this behavior by providing you own text, `[text](xref:...)`.
+
+:::{tip} References are stored in a [`myst.xref.json`](#myst-xref-json) file
+:class: dropdown
+All MyST sites published to the web expose a [`myst.xref.json`](#myst-xref-json) file that contains all of the cross-references information about a MyST site.
+This file is what MyST pulls when you point to an external MyST site for cross-referencing.
+It is also a machine-readable record that can be used for analyzing the cross-referencing behavior of MyST projects.
+:::
 
 (intersphinx)=
 
 ## Referencing external Sphinx documentation
 
 MyST can integrate directly with other Sphinx documentation, which is used in many Python projects including the [standard library](https://docs.python.org/).
 
-In your project configuration, include the `references` object with named links out to the Sphinx or MyST documentation that you will reference in the project. For example, in the demonstration below we will load the Python 3.7 documentation and JupyterBook docs, both of which use sphinx and expose cross references through an `objects.inv` file.
+Same as [MyST cross references](#myst-xref), use the `references` object to list Sphinx projects. For example, in the demonstration below we will load the Python 3.7 documentation and JupyterBook docs, both of which use sphinx and expose cross references through an `objects.inv` file.
 
 (intersphinx-config)=
 
@@ -48,8 +108,11 @@ references:
   jupyterbook: https://jupyterbook.org/en/stable/
 ```
 
-When you specify these in your project configuration, MyST will load the remote `objects.inv` file,
-and provide access to all of the references in that project. This `inv` file will be cached to disk locally in the `_build` folder, eliminating duplicate web requests on subsequent builds. If the target documentation updates and you must reload the remote references, you may delete the cache with `myst clean --cache`.
+```{tip}
+In your `references` object, you may freely mix MyST and Sphinx projects. MyST will decipher which to use, based on the presence of `myst.xref.json` or `objects.inv` files at the specified URL.
+```
+
+The behavior of these entries is identical to MyST cross references: the remote `objects.inv` file, which contains all available project references, is downloaded and cached in the `_build` folder.
 
 ````{important}
 # Intersphinx Examples
@@ -58,37 +121,32 @@ and provide access to all of the references in that project. This `inv` file wil
 :header-rows: 1
 * - MyST Syntax
   - Rendered
-* - `[](myst:python#zipapp-specifying-the-interpreter)`
+* - `[](xref:python#zipapp-specifying-the-interpreter)`
     : A reference to the reference documentation in Python.
-  - [](myst:python#zipapp-specifying-the-interpreter)
-* - `[](myst:jupyterbook#content:references)`
+  - [](xref:python#zipapp-specifying-the-interpreter)
+* - `[](xref:jupyterbook#content:references)`
     : A reference to the JupyterBook documentation, that brings you directly to the reference,
       as well as fills in the label text.
-  - [](myst:jupyterbook#content:references)
-* - `<myst:#library/abc>`
-    : A simplified link that will resolve to the first external inventory that satisfies the target.
-  - <myst:#library/abc>
+  - [](xref:jupyterbook#content:references)
 ```
 ````
 
-To reference a function, class or label in the linked documentation, use the `myst:` protocol in a link followed by the `project` key and the target.
-For example, `<myst:python#library/abc>` renders to:\
-"<myst:python#library/abc>"\
+To reference a function, class or label in the linked documentation, use the `xref:` protocol in a link followed by the `project` key and the target.
+For example, `<xref:python#library/abc>` renders to:\
+"<xref:python#library/abc>"\
 and is made of a `protocol`, `project` and `target`.
 
 protocol
-: The protocol for this type of link is `myst:`, and is what selects for cross-project referencing.
+: The protocol for this type of link is `xref:`, and is what selects for cross-project referencing.
 
 project
 : the `project` key above is "python" which is defined in your local [project configuration](#intersphinx-config) above.
-: The project is optional, however, we recommend that you include it to both efficiently look up the reference as well as be explicit as to what project you are referring to.
-: If the project is not included, all projects will be searched for the reference in the order given in the `intersphinx` configuration.
 
 target
 : The target is everything that follows the `#` and is a named reference in the project.
 : In the example above it is "library/abc".
 
-As with any link, the text can be overridden using markdown link syntax `[text](myst:...)`.
+As with any link, the text can be overridden using markdown link syntax `[text](xref:...)`.
 
 ````{tip}
 :class: dropdown
@@ -111,7 +169,7 @@ std:doc Abstract Base Classes for Containers (library/collections.abc)
   https://docs.python.org/3.7/library/collections.abc.html
 ```
 
-Use the target in the parenthesis, which would be `myst:python#library/abc` above.
+Use the target in the parenthesis, which would be `xref:python#library/abc` above.
 ````
 
 ## Wikipedia Links

docs/frontmatter.md

@@ -73,9 +73,6 @@ The following table lists the available frontmatter fields, a brief description
 * - `short_title`
   - a string (max 40 chars, see [](#titles))
   - page & project
-* - `name`
-  - a string (max 500 chars)
-  - page & project
 * - `description`
   - a string (max 500 chars)
   - page & project
@@ -85,6 +82,9 @@ The following table lists the available frontmatter fields, a brief description
 * - `downloads`
   - a download object, see [](./website-downloads.md)
   - page & project
+* - `label`
+  - a string (max 500 chars) to identify the page in cross-references
+  - page only
 * - `tags`
   - a list of strings
   - page only
@@ -98,7 +98,7 @@ The following table lists the available frontmatter fields, a brief description
   - a valid date formatted string
   - page can override project
 * - `authors`
-  - a list of author objects, see [](#authors)
+  - a list of author objects, see [](#frontmatter:authors)
   - page can override project
 * - `reviewers`
   - a list of author objects or string ids, see [](#other-contributors)
@@ -125,7 +125,7 @@ The following table lists the available frontmatter fields, a brief description
   - a string
   - page can override project
 * - `funding`
-  - a funding object, see [](#funding)
+  - a funding object, see [](#frontmatter:funding)
   - page can override project
 * - `github`
   - a valid GitHub URL or `owner/reponame`
@@ -245,7 +245,7 @@ banner: banner.png
 Example of a banner in a site using the `article-theme`.
 :::
 
-(authors)=
+(frontmatter:authors)=
 
 ## Authors
 
@@ -341,7 +341,7 @@ The `authors` field is a list of `author` objects. Available fields in the autho
 
 ### Reviewers, Editors, Funding Recipients
 
-Other contributors besides authors may be listed elsewhere in the frontmatter. These include `reviewers`, `editors`, and [funding](#funding) award `investigators` and `recipients`. For all of these fields, you may use a full [author object](#authors), or you may use the string `id` from an existing author object defined elsewhere in your frontmatter.
+Other contributors besides authors may be listed elsewhere in the frontmatter. These include `reviewers`, `editors`, and [funding](#frontmatter:funding) award `investigators` and `recipients`. For all of these fields, you may use a full [author object](#frontmatter:authors), or you may use the string `id` from an existing author object defined elsewhere in your frontmatter.
 
 (affiliations)=
 
@@ -623,6 +623,8 @@ String values for licenses should be a valid “Identifier” string from the [S
 
 By using the correct SPDX Identifier, your website will automatically use the appropriate icon for the license and link to the license definition.
 
+(frontmatter:funding)=
+
 ## Funding
 
 Funding frontmatter is able to contain multiple funding and open access statements, as well as award info.