HLS dynamic plugins loading proposal

[KovalevDima]

Is your enhancement request related to a problem? Please describe.

I’m often frustrated with the HLS plugin architecture for the following reasons:

1. It forces me to rebuild HLS every time I want to modify its plugin set.
2. It encourages me to override default package options to remove any plugin dependencies.
3. It restricts me from implementing my own plugins by requiring them to be pushed to the HLS monorepo.
4. It implies that everyone must compile a lot of dependencies with default plugin set

Describe the solution you'd like

In the cabal.project file, we can specify a list of packages that should be compiled and loaded into GHCi. For example:

extra-packages:
  hls-fourmolu-plugin
  hls-stylish-haskell-plugin
  hls-stan-plugin
  hls-floskell-plugin

AFAIK, HLS loads a GHCi session and has access to all of the project dependencies. If we modify the hls-plugin-api to:

1. Support discovering available plugins
2. Loading compiled plugins from GHCi session

then we will have a simple dynamic plugin loader that provides users with the following capabilities:

1. Defining their own plugins for local projects without modifying or redistributing HLS.

   -- cabal.project
   packages:
     ./myAwesomeProject
     ./hls-myAwesomeProject-plugin

2. Using all the HLS plugins available on Hackage without needing to recompile HLS.
3. Overriding any HLS plugins without recompiling HLS.

As a result, we will have a much more modular HLS architecture, which leaves the support of experimental or controversial plugins to the maintainers. It will also provide the ability to experiment with HLS with less effort. Additionally, it will simplify HLS distribution by making it independent from at least some of the existing plugins, reducing the heavy dependency footprint of HLS

Describe alternatives you've considered

I've been thinking about HLS recompilation before start. But I think it will increase HLS start time a lot

Additional context

Inspired by discussion #4464
I don't know if it is possible to implement, so please leave any feedback about this proposal