implement syntax highlighting

[swissiety]

LSP 3.16 is published with support for SemanticTokens.

• render syntax highlighting ranges. Starting point is the ExternalAnnotator LSPAnnotator.

[Trias]

I don't think that LSPAnnotator is the right starting point for syntax highlighting/semantic tokens. It's used to display warnings and such.

unfortunately syntax highlighting is so deeply integrated in intellij (PSI), i can't think of a way for us to use it.

If anything i think we have do reimplement all highlighting by ourselves, which may be quite an undertaking unfortunately. This shouldn't stop you from trying of course ;). If you have a more concrete idea on how to implement semantic tokens, I#m happy to hear it.

[nixel2007]

I think starting point should be SyntaxHighlighterBase.
May be language client may populate intellij with tokens got from semanticTokens request

[Trias]

@nixel2007 that was what i was referring to. For intelliJ syntax highlighting you need an (syncronous) incremental Lexer. It's a cool thing but very low level. i don't know how to make it work in combination with LSP. If you have more knowledge about this i would be happy to hear it.

[swissiety]

Via SyntaxHighlighterBase we need a lexer or a lexer emulating functionality which feeds Tokens into the SyntaxHighlighter - which is possible to emulate but it needs to wait for the semanticTokens Notification. Additionally it needs a lot of custom language specific config e.g. a way to determine the filetype at runtime - as the set of handled files is language server specific - by IntelliJ: suffix, path and content of the virtualFile is possible. Could be a way to do it.

My suggestion is to use an analog approach like for Annotations/Diagnostics. Implement the highlighting via editor.getMarkupModel().addHighlighter() / RangeHighlighter. Using TextAttributesKeys should allow for a color scheme config page, too.
That way we dont need a custom language implementation for lsp or even a faked lexer which would block the syntaxhighlighting until notified by semanticTokens (if it happens). Additionally the ExternalAnnotator (LSPAnnotator) is executed after syntaxhighlighting is finished.

[swissiety]

@nixel2007 that was what i was referring to. For intelliJ syntax highlighting you need an (syncronous) incremental Lexer. It's a cool thing but very low level. i don't know how to make it work in combination with LSP. If you have more knowledge about this i would be happy to hear it.

every SemanticToken Type would be a token of the emulated lexer. incremental lexer is supported but not necessary if i remember it correctly.