feat: definitions and references lsp hint #1016

rszyma · 2024-05-06T03:25:04Z

Describe your changes. Use imperative present tense.

Add goto definition support for LSP consumers. Adding support for it directly in LSP server could maybe be doable, but unreliable (e.g. templates are complicating things) and so a lot of parser logic would need to be reimplemented for all edge cases, so I think this is better added directly to kanata-parser.

For now I've added definition and references for aliases. I'd like to wait for recieve some feeback (i.e. if this is a direction we want to take) before adding the rest (variables, layers, etc.).

Currently the code is working for aliases when paired with rszyma/vscode-kanata#32

Implementation notes:

I couldn't add lsp_hints to ParserState, because it would require changing & to &mut on s param in many places which would be really ugly, and even then, when trying to switch to &mut at some point I got an error regarding multiple borrows of &mut s at the same time. So instead I've added additional lsp_hints param to many functions.

Other done things:

Move lsp_hint_inactive_code from ParserState to IntermediateConfig

Other thoughts:

Perhaps for performance purposes LSP features could be behind a feature flag, but it doesn't seem like it causes any major performance degradation + parser code runs only once, so for simplicity I didn't introduce the flag.

Checklist 1

Go-to definition and references implemented for:

Checklist 2

Add documentation to docs/config.adoc
- Yes or N/A
Add example and basic docs to cfg_samples/kanata.kbd
- Yes or N/A
Update error messages
- Yes or N/A
Added tests, or did manual testing
- Yes

rszyma · 2024-05-06T03:42:01Z

Also this would enable semantic highlighting I think?

jtroo · 2024-05-06T04:55:22Z

I couldn't add lsp_hints to ParserState, because it would require changing & to &mut on s param in many places which would be really ugly, and even then, when trying to switch to &mut at some point I got an error regarding multiple borrows of &mut s at the same time. So instead I've added additional lsp_hints param to many functions.

Consider using RefCell for this; I think it will suit the purpose well enough.

Perhaps for performance purposes LSP features could be behind a feature flag, but it doesn't seem like it causes any major performance degradation + parser code runs only once, so for simplicity I didn't introduce the flag.

I think we should do it for this new code and retroactively do it for the inactive code hints too. Probably good to compartmentalize the bulk of the new stuff into a new module.

We could make the functions always be called, and then the function body can be conditionally compiled (with an empty body by default); and rely on dead code elimination to do the right thing. Or add conditional compilation at the call-site to guarantee the optimization.

jtroo · 2024-05-07T05:03:27Z

parser/src/cfg/mod.rs

+ .push_from_atom(atom);
+ vkey_expr
+ .atom(s.vars())
+ .expect("must be atom since we're matching atom")


Fly-by comment

The way this new code works isn't right because matching an expr and finding an Atom might find you a $var that is actually a list.

You might want to add span_atom equivalent of span_list, which I haven't yet added because I didn't have the need for it.

Agree. I've refactored in 99be6a2. No need for span_atom

parser/src/cfg/sexpr.rs

This also fixes wasm build. Also simplify main.rs.

Also: - fix misleading error message when list after deflayermap token is empty. - capture only layer name in deflayer/deflayermap. Previously it captured span of a list where layer name is defined

rszyma · 2024-06-02T17:24:02Z

one problematic aspect of implementing defs/refs for variables is that there are 2 types of variables: normal (globally scoped) variables, and variables in deftemplate which have limited scope. deftemplate scoped variables shadow global ones?

Edit: I'd say let's just ignore existence of template variables from perspective of refs/defs.

rszyma · 2024-06-02T18:33:40Z

implementing for chord groups doesn't seem that important, but another cool thing about defs/refs hints is that it also will allow implementing semantic highlighting

Also adds "lsp" cargo feature. TODO: conditionally compile all lsp-related features behind this flag

rszyma · 2024-06-02T20:52:16Z

I've implemented defs/refs for variables.

SExpr.list and SExpr.atom don't have access to ParserState right now, so we can't access ParserState.lsp_hints though it too. We could add a lsp_hints parameter to it, but that requires 100+ changes and passing it in every list or atom call.

To minimize amount of changes and taking into consideration that it will be used only with "lsp" cargo flag, I think it's pretty reasonable to go for something like a global static mut variable.
One thing though, as I've mentioned in code comment, it makes parse_cfg_raw_string (and all majority of functions that are called in parse_cfg_raw_string) unsafe for concurrent calls. And so it's not a good idea to run tests with "lsp" flag on (we can't use "all" features flag in CI. Edit: I think we already don't?).

What do you think @jtroo?

Let's just ignore them for now, they don't seem to be important for goto definition. It can always be added later.

jtroo

I've implemented defs/refs for variables.
...
To minimize amount of changes and taking into consideration that it will be used only with "lsp" cargo flag, I think it's pretty reasonable to go for something like a global static mut variable.

The higher-level idea is sensible. The current implementation needs to be changed though; static mut is rarely if ever correct to do. The idea can be implemented without unsafe, and so should be. The parser is not multi-threaded, so one option is to use thread_local. Using thread_local also means parser+lsp tests should be able to be run in parallel just.

Although due to how str_to_oscode works, parser tests already shouldn't run in parallel anyway, so another option is to use Lazy<Mutex<...>> like how CUSTOM_STRS_TO_OSCODES does it.

rszyma · 2024-06-03T00:10:45Z

thread_local looks interesting, but I'm not sure if I fully understand it yet. Couldn't we also use thread_local in CUSTOM_STRS_TO_OSCODES to enable parrarel tests?

rszyma · 2024-06-03T00:59:41Z

The parser is not multi-threaded, so one option is to use thread_local

Done. Not sure if I'm using it correctly, but seems to work.

jtroo · 2024-06-03T02:20:21Z

thread_local looks interesting, but I'm not sure if I fully understand it yet. Couldn't we also use thread_local in CUSTOM_STRS_TO_OSCODES to enable parrarel tests?

I recall (perhaps incorrectly) that there are other threads other than the one the parser runs on for the real Kanata program. If I'm correct, using thread_local would mean the other threads do not see results from deflocalkeys, which would be incorrect behaviour.

jtroo

Needs a bit more cleanup and refining around conditional compilation; once that's done will be good to merge.

jtroo · 2024-06-03T04:45:21Z

parser/src/cfg/platform.rs

@@ -138,7 +138,7 @@ pub(crate) fn filter_env_specific_cfg(
 env_var_val.is_empty(),
 ) {
 (None, false) => {
- lsp_hint_inactive_code.push(LspHintInactiveCode {
+ lsp_hint_inactive_code.push(lsp_hints::InactiveCode {


Can we add the conditional compilation with LSP here as well?

parser/src/lsp_hints.rs

rszyma mentioned this pull request May 6, 2024

feat: go to definition rszyma/vscode-kanata#32

Merged

3 tasks

feat: definitions and references lsp hint - implement for aliases

7b0ff8d

rszyma force-pushed the definition-location branch from c2099f9 to 7b0ff8d Compare May 6, 2024 22:02

rszyma added 4 commits May 7, 2024 01:45

implement for layers

8312b88

implement for includes

a2528fa

refactor: move lsp hints to a separate module

f3f6139

implement for virtual keys + mini refactor to avoid duplicated code

df047ee

rszyma force-pushed the definition-location branch from f97183a to df047ee Compare May 7, 2024 02:51

fix an oopsie

1ae2202

jtroo reviewed May 7, 2024

View reviewed changes

parser/src/cfg/sexpr.rs Show resolved Hide resolved

rszyma added 4 commits June 2, 2024 02:12

Merge remote-tracking branch 'origin/main' into definition-location

7a0af21

fix: don't include @ symbol in alias name

ac32ab7

remove reduntant gui-related cfg directives

16604da

This also fixes wasm build. Also simplify main.rs.

Allow deflayermap without parantheses around name

0bfaa75

Also: - fix misleading error message when list after deflayermap token is empty. - capture only layer name in deflayer/deflayermap. Previously it captured span of a list where layer name is defined

implement for templates

590f2c0

implement for variables (RFC)

356d23c

Also adds "lsp" cargo feature. TODO: conditionally compile all lsp-related features behind this flag

rszyma marked this pull request as ready for review June 2, 2024 21:01

rszyma added 6 commits June 2, 2024 23:04

remove chord group from defs/refs

d066169

Let's just ignore them for now, they don't seem to be important for goto definition. It can always be added later.

small refactor

99be6a2

use extend(drain()) instead of clone_from`

d08ec9e

fix test cfg::tests::layer_name_allows_var

bd77921

move more code behind "lsp" cargo flag

25142b4

CI: add cargo clippy for parser with lsp feature

36596d3

jtroo reviewed Jun 2, 2024

View reviewed changes

use thread_local instead of unsafe static mut

51ea538

jtroo reviewed Jun 3, 2024

View reviewed changes

conditional compilation for lsp_hints.rs

0b1bf3d

jtroo approved these changes Jun 7, 2024

View reviewed changes

jtroo merged commit 306e172 into jtroo:main Jun 7, 2024
4 checks passed

rszyma deleted the definition-location branch June 7, 2024 18:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: definitions and references lsp hint #1016

feat: definitions and references lsp hint #1016

rszyma commented May 6, 2024 •

edited

Loading

rszyma commented May 6, 2024

jtroo commented May 6, 2024

jtroo May 7, 2024

rszyma Jun 2, 2024

rszyma commented Jun 2, 2024 •

edited

Loading

rszyma commented Jun 2, 2024

rszyma commented Jun 2, 2024 •

edited

Loading

jtroo left a comment

rszyma commented Jun 3, 2024

rszyma commented Jun 3, 2024

jtroo commented Jun 3, 2024 •

edited

Loading

jtroo left a comment

jtroo Jun 3, 2024

rszyma Jun 3, 2024

feat: definitions and references lsp hint #1016

feat: definitions and references lsp hint #1016

Conversation

rszyma commented May 6, 2024 • edited Loading

Describe your changes. Use imperative present tense.

Checklist 1

Checklist 2

rszyma commented May 6, 2024

jtroo commented May 6, 2024

jtroo May 7, 2024

Choose a reason for hiding this comment

rszyma Jun 2, 2024

Choose a reason for hiding this comment

rszyma commented Jun 2, 2024 • edited Loading

rszyma commented Jun 2, 2024

rszyma commented Jun 2, 2024 • edited Loading

jtroo left a comment

Choose a reason for hiding this comment

rszyma commented Jun 3, 2024

rszyma commented Jun 3, 2024

jtroo commented Jun 3, 2024 • edited Loading

jtroo left a comment

Choose a reason for hiding this comment

jtroo Jun 3, 2024

Choose a reason for hiding this comment

rszyma Jun 3, 2024

Choose a reason for hiding this comment

rszyma commented May 6, 2024 •

edited

Loading

rszyma commented Jun 2, 2024 •

edited

Loading

rszyma commented Jun 2, 2024 •

edited

Loading

jtroo commented Jun 3, 2024 •

edited

Loading