Technical design principles

[Felienne]

We have lovely design goals for the language that help in decision making when we are in doubt between different options, but we might also needs some principles guiding the larger project and the implementation itself.

We already made some decisions that we might want to keep going forward, for example:

TG1 We want to be (and remain) technically ready to support several natural languages

The reason that we rejected (#340) in essence was that it would make it a lot harder to implement multiple languages in that way.

Let's discuss any other technical goals that will help us in decision making here.

[Felienne]

A decision we did not make explicitly but I think is worth adding is about the trade off between putting stuff in the grammar or in subsequent processing of the grammar. I just made #804 in which I do a lot of magic in the Python code that also (partly) could be done in the grammar. That did not feel great, so I think we want to put stuff in the grammar when possible:

TG2 When in doubt, put logic in the grammar

My reasoning for this is that grammars are quite universal and it requires less reading/reconstructing of mental models than interacting with the code, so it might lower the threshold for participation.

[Felienne]

This TG actual helped me in #843 because I was in doubt where to add the logic and I updated it the grammar for this reason!

Actually, the incomplete exception could be integrated into the grammar even more by having separate rules for incomplete commands:

print: _PRINT (_SPACE text)? and print_incomplete: _PRINT

It might make parsing a bit slower (because of overlapping rules) but it saves lines of code in processing the AST and might save time there.

[bjorn3]

Rust-analyzer uses a hand written parser for rust to allow recovery from even completely nonsensical code in most cases. It outputs a CST (concrete syntax tree) using rowan instead of an AST. A CST is basically an AST except that every token is preserved to allow for lossless parsing. It also allows any part of the tree to be missing, like the print argument in this case. @matklad gives an explanation of how rust-analyzers parser does error recovery in https://m.youtube.com/watch?v=0HlrqwLjCxA&list=PLhb66M_x9UmrqXhQuIpWC5VgTdrGxMx3y&index=15 (honestly haven't watched it myself yet) It may be too big of a project to do for now to rewrite the parser into a manual recursive descent parser, but it is something to keep in mind. Using a custom parser framework written with gradual changes and parser recovery in mind would also be an option I think, though probably an even bigger project.