Can we write EBNF for djot?

[shi-yan]

I have a n00b question that if we can write EBNF for djot.

I want to use djot as the file format for my notebook app. But I need to extend djot's syntax to support more features. I looked at some of the djot implementations, they are too obscured to modify. I hence wanted to write one from scratch.

I'm not experienced in parsing. Previously I have used some parser generators briefly. I understand one design goal of djot is making parsing easier, I'm wondering if we can write EBNF for djot? (if so I will consider using tree-sitter to create a parser)

thanks,

[matklad]

Practically, my gut feeling is that a grammar won’t be very useful for djot (or in general for light markup languages). The structure is much more shallow (few distinct syntactic constructs), any input should have a parse tree, and there are a bunch of 2D rules (indentation, block quotes). So I’d probably recommend doing what djot.JS implementation does (parse blocks using sort-of-recusive descent maintaining a stack of blocks, witching each block parse inlines maintaining a map of potential openners).

Theoretically, I think djot isn’t LL(k) (because of attributes and rules for emphasis), but it might be LR or PEG.

But I need to extend djot's syntax to support more features

Could you tell which features exactly do you need? One big idea of djot is that you shouldn’t need to extend its syntax, as div, spans, and attributes should be enough to express any custom functionality within djot’s syntax.

[matklad]

Thanks for the example! I think everything you want can be achieved using existing djot syntax and doing customization via filters & overrides machinery. For example, we can start with the following djot document:

# This is a test title
: tags: science, math

This is a you tube video: <youtube:RvRhUHTV_8k>

And a github project card:

<github:matklad/rust-analyzer>

An equation: $$`x^n + y^n = z^n`{#fermat}

We can refer to it as <eq:fermat>

Everything here is standard djot syntax, which is parsed into a normal djot AST:

{
  "tag": "doc",
  "references": {
    "This is a test title": {
      "tag": "reference",
      "label": "This is a test title",
      "destination": "#This-is-a-test-title"
    }
  },
  "footnotes": {},
  "children": [
    {
      "tag": "section",
      "children": [
        {
          "tag": "heading",
          "level": 1,
          "children": [
            {
              "tag": "str",
              "text": "This is a test title"
            }
          ]
        },
        {
          "tag": "definition_list",
          "children": [
            {
              "tag": "definition_list_item",
              "children": [
                {
                  "tag": "term",
                  "children": [
                    {
                      "tag": "str",
                      "text": "tags: science, math"
                    }
                  ]
                },
                {
                  "tag": "definition",
                  "children": []
                }
              ]
            }
          ]
        },
        {
          "tag": "para",
          "children": [
            {
              "tag": "str",
              "text": "This is a you tube video: "
            },
            {
              "tag": "url",
              "text": "youtube:RvRhUHTV_8k"
            }
          ]
        },
        {
          "tag": "para",
          "children": [
            {
              "tag": "str",
              "text": "And a github project card:"
            }
          ]
        },
        {
          "tag": "para",
          "children": [
            {
              "tag": "url",
              "text": "github:matklad/rust-analyzer"
            }
          ]
        },
        {
          "tag": "para",
          "children": [
            {
              "tag": "str",
              "text": "An equation: "
            },
            {
              "tag": "display_math",
              "text": "x^n + y^n = z^n",
              "attributes": {
                "id": "fermat"
              }
            }
          ]
        },
        {
          "tag": "para",
          "children": [
            {
              "tag": "str",
              "text": "We can refer to it as "
            },
            {
              "tag": "url",
              "text": "eq:fermat"
            }
          ]
        }
      ],
      "attributes": {
        "id": "This-is-a-test-title"
      }
    }
  ]
}

in your project, you can give additional semantics to specific shapes of the AST. For example, every url which starts with youtube: would be rendered as youtube's iframe, and every eq: as a numbered linked to that equation. Similarly, you can extract tags from the definition list just after the top-level header.

If you are using djot.js, you'll probably use filters & overrides infrastructure for that:

• https://github.com/jgm/djot.js#filters
• https://github.com/jgm/djot.js#rendering-the-djot-ast-to-html

This probably isn't as nice as having first class syntax for these concepts, but should be nicer than having to write your own parser!

[matklad]

As a somewhat non-trivial example of using overrides, infrastructe, see, for example, my blog, where I define things like admonitions or <kbd> input:

https://github.com/matklad/matklad.github.io/blob/486c16abf201d03abb7a8c5b3da939d278316a73/djot.ts#L30

[shi-yan]

Thank you so much! This sounds wonderful. I will look into it!

[hellux]

The https://github.com/hellux/jotdown implementation should also support usecases like these. If you need an AST it is quite easy to build it, but you could possibly just work with the events produced from the document. For example, the numbering of equations can be implemented by simply modifying the link events, e.g:

use jotdown::Container::{Link, Math};
use jotdown::Event::{Start, Str};
use jotdown::LinkType::AutoLink;
use jotdown::Render;

fn main() {
    let input = "
        $$`a = b + c`{#abc}

        $$`d = e + f`{#def}

        See equation <eq:abc> and <eq:def>.
    ";

    let mut equations = std::collections::HashMap::new();
    let mut num_eq = 0;
    let mut eq_ref = None;
    let events = jotdown::Parser::new(input).map(|ev| match ev {
        Start(Math { display }, attrs) => {
            if let Some(id) = attrs.get("id") {
                num_eq += 1;
                equations.insert(id.to_string(), num_eq);
            }
            Start(Math { display }, attrs)
        }
        Start(Link(url, AutoLink), attrs) => Start(
            Link(
                if let Some(eq) = url.strip_prefix("eq:") {
                    eq_ref = Some(equations.get(eq).copied().unwrap_or(-1));
                    format!("#{eq}").into()
                } else {
                    url
                },
                AutoLink,
            ),
            attrs,
        ),
        Str(..) if eq_ref.is_some() => Str(format!("({})", eq_ref.take().unwrap()).into()),
        _ => ev,
    });

    let mut html = String::new();
    jotdown::html::Renderer.push(events, &mut html).unwrap();

    assert_eq!(
        html,
        concat!(
            "<p><span id=\"abc\" class=\"math display\">\\[a = b + c\\]</span></p>\n",
            "<p><span id=\"def\" class=\"math display\">\\[d = e + f\\]</span></p>\n",
            "<p>See equation <a href=\"#abc\">(1)</a> and <a href=\"#def\">(2)</a>.</p>\n"
        ),
    );
}

[shi-yan]

Thank you so much for this example! My first attempt was actually hacking jotdown, as I previously modified pulldown to add new syntax.

But I wasn't patient enough to make much progress in understanding jotdown's code. If I can simply provide the AST, that will be awesome.