New language frontend merging & architectural change

[inclyc]

Motivation

For now controllers just send full text file to workers, and periodically killing workers, if they are outdated.
This is very error-prone because the controller parses the file for location information, or attrpath construction, and workers parses these files again.
Previously we don't have a memory-safe nix parser and it is coupled with evaluators in upstream C++ nix. Thus the only solution is sending raw text contents to the worker, and let them evaluate the file, the controller just roughly redirect lsp requests to workers. And since the controller does not have a good parser, we have experienced very high overhead.

Related issues:

• options suggestion is not working without indentation #277
• Cannot follow home-manager example #274
• can't run example #271
• rename function with default parameters not works properly #255

Previous Work

There are a parser generated by bison and in-complete semantic analysis module in these two PRs. For now they are outdated because we have a new handwritten parser.

• nixd/Syntax: init #257
• nixd/Sema: the semantic analysis for our new parser #260

The iteration plan

Let the controller do all parsing stuff, and serialize ASTs to workers.

As commented above, currently both controller process & worker process do parsing stuff, and I think a better architecture is just offload all parsing job to the controller. Serialize ASTs in shared memory and let workers fetch the AST and perform evaluation.

The controller will ask worker process values & envs for specific nodes from worker, not sending full text contents. The serialization format & implementation:

• libnixt: add Serialize #276

Uses a new language frontend just for parsing, but uses official nix implementation for evaluation

This is a technical decision. Because evaluator is not very easy to maintain and current implementation "just works" for language servers. Here are PRs related to the new frontend:

Misc Refactor

• libnixf: add to top-level build #278
• libnixf: remove green node design #291
• libnixf: rename getter for Token , getXxx() -> xxx() #293
• libnixf: line column range #296
• libnixf: remove fmt dependency #298
• libnixf: flatten libnixf/lib/meson.build, to a single library #300
• libnixf: use unique_ptr #326
• libnixf/Parse: create a class for Parser, expose testing API #328

Parser

string_part : interpolation
            | STRING_PART
            | STRING_ESCAPE
interpolation : "${" expr "}"
string : " string_part* "
       | '' string_part* ''

• libnixf: parse ExprString #292
• libnixf: parse string interpolation #294
• libnixf: parse indented string #295

 path : path_fragment (path_fragment)*  path_end
Context     PS_Expr       PS_Path        PS_Path

• libnixf: parse ExprPath #297

'(' expr ')'

• libnixf: parse ExprParen ( expr ) #308

attrset_expr : REC? '{' binds '}'

binds : ( binding | inherit )*

inherit :  'inherit' '(' expr ')' inherited_attrs ';'
        |  'inherit' inherited_attrs ';'

inherited_attrs: attrname*

binding : attrpath '=' expr ';'

attrpath : attrname ('.' attrname)*

• libnixf: parse ExprAttrs (basic, without inherit support) #314
• libnixf: parse inherit binding #321

expr_simple :  INT
            | FLOAT
            | string
            | path
            | hpath
            | URI
            | expr_paren
            | legacy_let
            | attrset_expr
            | list_expr

hpath : "~" prefixed path, $HOME hack

• libnixf: parse variable #315
• libnixf: use sync tokens for creating unknown nodes (error recovery) #316

expr_select : expr_simple '.' attrpath
            | expr_simple '.' attrpath 'or' expr_select
            | expr_simple 'or' <-- special "apply", 'or' is argument
            | expr_simple

• libnixf/Parse: parse expr_select #329

expr_app : expr_app expr_select
         | expr_select

• libnixf: parse expr_app #330

list_expr : '[' expr_select* ']'

• libnixf: parse list_expr #331

if_expr : 'if' expr 'then' expr 'else' expr

• libnixf/Parse: parse expr_if #349

assert_expr : 'assert' expr ';' expr

• libnixf/Parse: parse expr_assert #350

let_in_expr :  'let' binds 'in' expr

• libnixf/Parse: parse expr_let #351

legacy_let: `let` `{` binds `}`

• Parse Legacy Let

with_expr :  'with' expr ';' expr

• libnixf/Parse: parse expr_with #354

expr_op : '!' expr_op
        | '-' expr_op
        | expr_op BINARY_OP expr_op
        | expr_app

• libnixf/Parse: parse expr_op #347

lambda_expr : lambda_arg ':' expr

lambda_arg : ID
           | ID @ {' formals '}'
           | '{' formals '}'
           | '{' formals '}' @ ID

formals : formal? (',' formal)*

formal : ID
       | ID '?' expr
       | "..."

• libnixf: parse lambda_arg #338
• libnixf: parse lambda_expr #339

expr      : lambda_expr
          | assert_expr
          | with_expr
          | let_in_expr
          | if_expr
          | expr_op

• Parse Expr

Semantic Lowering

• libnixf: add semantic analysis for AttrSets #323
• libnixf/Sema: lowering inherit #333
• libnixf/Sema: semantic lowering for lambda formals #340

Semantic Checking

• libnixf: variable lookups #372
• Various Static Linting

The new worker, and new controller

Controller

• nixd-next: init, without any capabilities #301
• nixd-next: text document synchronization #303
• nixd-next: basic diagnostics support in the Controller #307
• nixd-next: add diagnostic quick fix (code action) #310
• nixd: support hover #318
• Document Link

nix-node-eval

• WIP

[inclyc]

All frontend things done. Waiting for #421 for lsp features impl