This document describes a set of tools for formatting RFCs, Internet Drafts, and other documentation, collectively called “rfcdoc”. These tools simplify the creation of documents by allowing authors to write in plain text files instead of hand-editing XML files. Also files like YANG modules and example content files are kept in distinct files to allow validation of their content and allow editors to enable mode-specific features. The files are then combined via commands triggered by “make” to generate complete RFCs and Internet-Drafts.
In the NETCONF/NETMOD working groups, we’ve been using some simple tools to ease the production of Internet Drafts and RFCs. I use the same tools for various open source projects (libxo, libslax, juise). The goals are simple:
- Humans should never write XML; it’s a fine language for machine interchange, for making people write it directly is just plain mean.
- Text should be email friendly. We all use email to talk/review, so having non-text formats make no sense.
- There are existing text conventions (even twenty plus years ago when I first wrote these tools) to allow semantic formatting.
- When they aren’t, we can roll-our-own easily enough.
With those simple goals, let me introduce our tools.
If you lack the patience to weed through all these details, feel free to skip directly to the “how to” instructions locatated in ^how-to^.
“oxtradoc” is the main tool, which turns text into “xml2rfc”-style XML content. “oxtradoc” stands for “outputs xml to rfc and documentation”. Not the best name, but at least I didn’t call it “doccer”.
Full documentation on oxtradoc’s formatting features is available in ^oxtradoc^.
“xml2rfc” is a python script that turns XML of a certain format into RFC-style text. It’s very good at what it does and I’m so glad I didn’t have to write it.
By far the most pertinent fact about xml2rfc here is that we don’t ever need to hand-write XML content for it. It is used to process XML generated by oxtradoc. See also:
https://pypi.python.org/pypi/xml2rfc https://xml2rfc.tools.ietf.org/public/rfc/html/rfc2629.html
Everyone’s favorite open-source YANG compiler, with a plug-in facility that allows various output targets. For details, see the following link:
https://github.com/mbj4668/pyang
“make” is the glue that holds it all together. The file “rfcdoc.mk” spells out the dependencies and commands that get the tools invoked correctly.
There are “make” targets for producing the final documents, as well as checking the content of YANG modules and example data files. “YANG tree” files can also be automatically generated.
Together these tools allow RFC authors to simplify their workload, be more productive, make fewer mistakes, and generate more accurate and useful drafts.
“oxtradoc” performs the translation from plain text into xml2rfc-style XML content. Specific rules must be followed, but the text input is generally readable and simple to understand.
Emacs has a mode called “outline mode”, which allows organization of text. “outline mode” has numerous functions for displaying and traversing these hierachies. See ^https://www.emacswiki.org/emacs/OutlineMode^ for details.
Another mode called “org mode” extends this mode with additional formatting conventions. “org mode” is a super set of “outline mode” and can be used similarly. See ^http://orgmode.org^ for details.
- Headings are in Emacs “outline mode”, where heading depth is
indicated a leading series of asterisks (“*”). The more asteriskss, the deeper the heading.
- History
** Early History *** History That Time Forgot ** Middle History ** Late History *** Missing History **** Dog Ate My History
- References can be make to URLs, anchors (headings), or external
references. A reference uses the convention of the target surrounded by up-hats:
The source of the Nile (^NILESOURCE^) was unknown (^NILEHISTORY^).
- Headings can be used as the anchors. An explicit anchor name can be
provided by placing the name at the end of the heading line between “at” signs (“@”), e.g. “@anchor-name@”.
*** Possible Sources of the Nile @NILESOURCE@
- If no explicit name is given, the anchor name will be the heading
content with the following conversions:
- Upper case is translation into lower case.
- Spaces are translated into dashes (“-“).
- Any non-alphanumeric characters besides dash and underscore are removed.
- A reference to an internal anchor is translated to the section
number of the anchor’s heading, prefixed with the word “Section”, e.g. “Section 4.2.1”. To suppress the prefix, use dual uphats:
See sections ^^ref-one^^ and ^^ref-two^^.
- Paragraphs of text are separated by blank links, just like one types
normally.
We are met on a great battlefield of that war. We have come to dedicate a portion of that field as a final resting place for those who here gave their lives that that nation might live.
It is altogether fitting and proper that we should do this. But in a larger sense we can not dedicate – we can not consecrate – we can not hallow – this ground. The brave men living and dead who struggled here have consecrated it far above our poor power to add or detract.
The world will little note nor long remember what we say here but it can never forget what they did here. It is for us the living rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced
Figures, XML, code, and ASCII art can be placed directly in the file by indenting the content. When text is intended, the other rules of formatting are not applied. While “oxtradoc” can detect the “type” of the content (in terms of XML, code, artwork, etc), any such content is wrapped in the xml2rfc “figure” and “artwork” tags, and is not formatted.
Example:
---- ------ -------------
| me | –>(emacs)–> | text | –>(oxtradoc)–> | documentation |
---- ------ -------------
Example:
<rpc> <order-lunch> <togo/> <pizza> <size><large/></size> <topping><sausage/></topping> </pizza> </order-lunch> </rpc>
Note that there is no need to hand-escape less-than and greater-than characters; “oxtradoc” does this automatically.
Figures can be placed in external files and included using the filename using a line containing “!!include-figure”:
!!include-figure ex-content.xml
An “extract-to” value can be added to indicate a desired extraction filename:
!!include-figure my.yang extract-to=”my-@2017-04-01.yang”
Putting such content in external files allows the content to be validated, as well as allowing text editors to apply mode-specific behaviors such as intentation or colored display.
“oxtradoc” supports a number of types of lists, based on simple text encoding conventions.
A dash (“-“) at the beginning of the line marks a symbol list item, where the item will be prefixed with a symbol.
Example::
Favorite Things:
- Raindrops on roses
- Whiskers on kittens
Formatted output::
Favorite Things:
- Raindrops on roses
- Whiskers on kittens
An underscore (“_”) at the beginning of the line marks an empty list item, where the item will not be prefixed with a symbol. This allows the lines to be adjacent without being wrapped together.
Example::
_ Do not pass Go _ Do not collect $200
Formatted output::
_ Do not pass Go _ Do not collect $200
A plus sign (“+”) at the beginning of the line marks a numbered list item, where the item is prefixed with an increasing integer number.
Example::
- The money
- The show
- Get ready
- Go, Cat, Go!
Formatted output::
- The money
- The show
- Get ready
- Go, Cat, Go!
An equals sign (“=”) at the beginning of the line marks an hanging list item, where the rest of the line is the hang text and the following lines are the content.
Example:: = 64-bit Machines that have 64-bit CPUs = 32-bit Machines that have 32-bit CPUs = 8-bit Machines that are really, really old
Formatted output::
= 64-bit Machines that have 64-bit CPUs = 32-bit Machines that have 32-bit CPUs = 8-bit Machines that are really, really old
oxtradoc using “org mode” tables. These tables are simple, visually accurate, and can be automatically generated using Emacs tools. See ^http://orgmode.org/guide/Tables.html^ for additional information.
The rules for these tables are fairly simple:
- Tables start with a vertical bar (“|”).
- Table headers appear next, separated by vertical bars.
- A separator line follows, containing vertical bars and plus signs.
- Content follows, separated by vertical bars.
- If a title is needed, the table is preceeded by a line containing
two dashes, the title, and an optional anchor:
Example::
– List of Sub-Statements @sub-list@
| substatement | section | cardinality |
|---|---|---|
| argument | ^extension-arg^ | 0..1 |
| description | ^description^ | 0..1 |
| reference | ^reference^ | 0..1 |
| status | ^status^ | 0..1 |
Formatted output::
– List of Sub-Statements @sub-list@
| substatement | section | cardinality |
|---|---|---|
| argument | ^fake-ref^ | 0..1 |
| description | ^fake-ref^ | 0..1 |
| reference | ^fake-ref^ | 0..1 |
| status | ^fake-ref^ | 0..1 |
Digressions are sub-blocks of texts that don’t appear in the RFC. There are three types of digressions: document blocks, references blocks, and open questions.
A digress begins with a line that starts with two open braces (“{{“) followed by the name of the block and a colon (“:”). The digress ends with a line containing two close braces (“}}”).
{{type-name: contents of the digress }}
Digressions of unknown type are ignored completely.
? Note that this needs to be updated for RFC7991, which deprecates some of these fields.
The “document” digression contains information about the document and is used for RFC header fields. The contents of the digression are name/value pairs on lines ending with semi-colons:
{{document: ipr pre5378Trust200902; category std; references back.xml; abbreviation YANG; title “The YANG 12.0 Data Modeling Language”; contributor “author:Joe Example:Example Inc:j@example.com”; keyword NETCONF; keyword XML; keyword “data modeling”; }}
Document values are taken from both the document section and the arguments to oxtradoc, with the latter taking precedence. The following section lists the name, options, and contents of the variables fields within the document digression:
= name (-n) Name of the document, suitable for rfc/@docName. = ipr (-i) (default “none”) IPR classification, suitable for rfc/@ipr. = category (-c) Category of the document, suitable for rfc/@category. = references Lists a file containing the references, in XML format. See ^ref-back^ for additional information. = abbreviation (-a) Short name of the document, suitable for rfc/front/title/@abbrev. = title (-t) Formal title of the document, suitable for rfc/front/title. = contributor Lists information about a document contributor, and may appear multiple times. Each value should be a role, name, organization, and email address separated by colons. = keyword Keywords for the document, suitable for rfc/front/keyword. This line may appear multiple times to supply a set of keywords.
A “open question” digress is used to record an open question within the document for the benefit of the document authors. The question should be resolved before publication. An open question can be entered using a “question” digression, or may appear in a line that starts with a question mark.
? Are we sure we want to encode this in EBCDIC?
External references are handled by oxtradoc using a two-phase approach. Typically a “refences.txt” file is created by the document author that lists the references used in the document. The digression can appear directly in the document, but there is some significant processing involved in turning this simple list into the complex and verbose XML that xml2rfc needs for references. “oxtradoc -m mkback” turns a references file (“references.txt”) into a suitable XML file (“references.xml”). The Makefile for rfcdoc handles this dependency automatically.
The format of the references file is a digression with the name “references”. The digression contains a “title” and a series of reference names. The title for a references digression can be “Normative References” or “Informative References”:
{{references: title “Informative References”; ietf-ref RFC2119; ietf-ref RFC6241; ietf-ref I-D.ietf-netmod-revised-datastores; }}
- Lines starting with pound signs (“#”) are comments and are
ignored.
- Lines starting with at signs (“@”) are passed through upchanged.
This serves as an escape mechanism for avoiding oxtradoc processing or for allowing pre-formatted content. Consider it as an option of last resort.
Since github.com supports “org mode” rendering of “.org” files, naming your file with a “.org” extension means automatic rendering of a subset of oxtradoc features in the github file display.
This project consists of two github repositories. The first (“rfcdoc”) contains the captive versions of the tools used (oxtradoc, pyang, xml2rfc) and a small .mk file that simplifies using the tools.
The second (“rfcexample”) is a fork-friendly repository that simplifies the use of rfcdoc by allowing the user to fork that repo and immediately start using the tools. Use of “rfcdoc” is not tied to “rfcexample”, but it might be easier for beginners.
“rfcexample” includes a Makefile that gives access to variables and targets that can be invoked from the command line to perform formatting, validation, and other tasks.
The “Makefile” in your project will direct the operation of these tools. The Makefile should have the following fields:
= draft Filename of the document = output_base Basename (filename without extension) of the output file. = examples Lists any example files. = trees Lists files of tree diagrams that should be automatically generated from the YANG files. = std_yang Lists YANG modules that are part of the document. = ex_yang = references_src Source filename for the references, e.g. “references.txt”. = references_xml Destination filename of the references, e.g.”references.xml”. = rfcdoc Directory were the “rfcdoc” git module appears.
In this example Makefile, the user has written a draft named “nmda-netconf.org”, which is used to generate “draft-dsdt-nmda-netconf-XX.txt”. A YANG module named “ietf-netconf-datastores.yang” is included as part of the generated file, and can be validated via the “make validate” command.
draft = nmda-netconf.org output_base = draft-dsdt-nmda-netconf examples = trees = std_yang = ietf-netconf-datastores.yang ex_yang =
rfcdoc = rfcdoc include ${rfcdoc}/rfcdoc.mk
When working with multiple drafts, a common rfcdoc/ directory can be shared, typically by cloning the “rfcdoc” project into the parent directory, rather can forking the “rfcexample” project. To use this ../rfcdoc/ directory, set the rfcdoc variable to that path in your Makefile:
rfcdoc = ../rfcdoc
The following targets are available for “make” via rfcdoc.mk:
= submit “make submit” will build both the text and xml versions of the draft. = validate Validates all YANG modules using pyang, include both standard modules and example modules. Any XML example payloads are also validated. = idnits Runs the “idnits” program on the draft so identify RFC-editor-related issues before publication. For example, references in the “Abstract” are not allowed. = new-tag After submitting the draft for publication, use this target to advance the number of the draft, e.g. from “-00.txt” to “-01.txt”. We use a “git tag” for this. = clean Cleans all generated files from the directory.
This section contains the simple steps for building your RFC. In these examples, replace MYNAME with your github login and MYDRAFT with that name of your draft.
Avoid the desire to use the words “draft” or your own name in the name of your draft, since this will hopefully be temporary and your draft will be accepted by a working group, and eventually published. Once it’s an RFC, having the base document be called “draft-phil-magic-cookies.org” would be bad. Best to start naming it “magic-cookies.org” from the start.
= Fork the rfcexample repo Visit the https://github.com/philshafer/rfcexample and click on the “Fork” button in the upper right.
= Rename the repo to something appropriate Under “Settings”, enter the new name for your repo, MYDRAFT. If you want to use github to track issues with your draft (a fine idea), the select “Issues” under the “Features” list. Then click on the “Rename” button.
= Clone the rfcexample repo On your laptop, make a local copy of the repo (aka “clone”) by using the “git clone” command. Note that the need the “–recursive” flag to ensure that the submodules rfcdoc uses are fully populated.
git clone –recursive https://github.com/MYNAME/MYDRAFT cd MYDRAFT
= Build the rfcdoc repo The rfcexample repo (and now your MYDRAFT repo) contains a set of submodules containing captive copies of the required tools. This is for simplicity and ease-of-use, but also keeps things stable. You need to build these tools using the following command:
sh rfcdoc/install.sh
Resolve any errors before continuing any further. If the tools are not installed correctly, nothing will work, so be sure this step is done properly.
= Rename the starting point document Your repo has a copy of a document meant to help you get started, but you’ll need to rename it before continuing:
git mv rfcexample.org MYDRAFT.org git commit -m ‘Rename’ MYDRAFT.org git push
= Edit your Makefile The Makefile under your new repo has a few variables that direct the operation of the tools. The first two are vital, and the rest can be used as your document grows. The “draft” variable is the name of your input document, so it should be set to “MYDRAFT.org”. The output_base is the basename of your output document, and that value will depend on the working group, status, and topic of your document. For now, we’ll use “draft-MYNAME-MYDRAFT”. The remaining variables should be empty. The last two lines of the file should not be changed, since they allow us to find your copy of rfcdoc.
draft = MYDRAFT.org output_base = draft-MYNAME-MYDRAFT examples = trees = std_yang = ex_yang =
rfcdoc = rfcdoc include ${rfcdoc}/rfcdoc.mk
= Test the build To performa a test build, run “make”. It should make a fairly meaningless document, but that should fill your eyes with pride.
= Edit your README.md Fix your README.md to contain proper and accurate information.
= Start editing your draft Go on! Get to work!
{{document: name ; ipr noModificationTrust200811; category info; title “rfcdoc: tools for building RFC and Internet-Drafts”; contributor “author:Phil Shafer:Juniper Networks:phil@juniper.net”; }}