Create distributable cross-platform packages #329

marvinborner · 2023-12-03T20:06:30Z

Most operating systems (e.g. GNU/Linux, BSDs) treat all text after the first space as a single argument. When using env in a script it is thus not possible to specify multiple arguments.[1]

Without this change the invocation of effekt after sbt install results in the following error:

/usr/bin/env: ‘java -jar’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines

MacOS/FreeBSD's env apparently behaves differently, so please verify that the resulting binary still works on them.

[1]: GNU coreutils documentation

b-studios · 2023-12-03T20:28:04Z

Hey this one line has a very long history.

We discussed it here at some point:
#16

Introduced env: 69655ae#diff-5634c415cd8c8504fdb973a3ed092300b43c4b8fc1e184f7249eb29a55511f91R85
Removed env: 3177944
Reintroduced env: 234314f
Added -S: 71f2ca2
Removed -S: 7834aed

It seems we cannot find a version that works on all platforms, so why not generate platform specific versions? Or look into https://github.com/xerial/sbt-pack ?

marvinborner · 2023-12-03T21:00:14Z

Oh, sorry for not noticing the history.

I just tried a JAR wrapper from StackOverflow which seems to work (at least for me) and bypasses the env portability issues. I'm not sure about Windows, but the previous version would've only worked with bash emulation (like cygwin) anyway, right?

I suppose an official solution like sbt-pack would be much cleaner.

Edit: The -S flag of env was released with coreutils 8.30 in 2018, so I don't really understand how it didn't work on Ubuntu 22.04.

jiribenes · 2023-12-03T23:02:02Z

Personally, I use something like:

#! /usr/bin/env bash -e

# ... add node to PATH ...

exec long/full/path/to/bin/java -jar another/long/path/to/effekt.jar "$@"

for my version of effekt.sh both on MacOS and on Linux without any problems -- I never use the shebang on the jar file :)

(automatically generated here: https://github.com/jiribenes/effekt-nix/blob/3147a32bce206520a6d562a2469638baa9048722/effekt.nix#L24-L27; just posting for archival reasons)

However, I believe it may be wise to explore sbt pack linked above further. :)

b-studios · 2023-12-03T23:08:26Z

Maybe at some point we can also look into native image again (https://www.graalvm.org/latest/reference-manual/native-image)

So far, I never went through the troubles to set it up in CI and have it automatically build binaries for the different platforms.

marvinborner · 2023-12-04T15:16:00Z

I did some experiments with sbt-scala-native (chosen over sbt-native-image since it has official support by sbt-crossproject).

Since I'm still unfamiliar with sbt/scala I have some questions:

What should we set in .nativeSettings? Is leaving it empty okay for now?
I get a weird error if I try effektNative/compile or something similar:

[error] -- Error: .../kiama/shared/src/main/scala/kiama/util/Source.scala:101:11
[error] 101 |case class FileSource(name: String, encoding: String = "UTF-8") extends Source {
[error]     |           ^
[error]     |   class FileSource does not contain field kiama$util$Source$$$1$$lzy2
[error] -- Error: .../kiama/shared/src/main/scala/kiama/util/Source.scala:96:11
[error] 96 |case class StringSource(content: String, name: String = "") extends Source
[error]    |           ^
[error]    |   class StringSource does not contain field kiama$util$Source$$$1$$lzy1
[error] two errors found
[error] (kiamaNative / Compile / compileIncremental) Compilation failed

Am I missing something obvious?

b-studios · 2023-12-04T21:05:44Z

I am not sure whether Scala Native is in a state where we can just compile our compiler using it :(

marvinborner · 2023-12-06T12:51:06Z

Small update: GraalVM native-image sbt effektJVM/nativeImage generates a binary file that seems to work flawlessly. I did not need to use any reflection files or similar.

b-studios · 2023-12-06T12:52:11Z

Cool! Can you try to use it with the vscode plugin as a language server?

Ah, and the Repl of course

marvinborner · 2023-12-06T14:12:09Z

One problem we need to solve is including the standard library in the native file. For now we need to manually specify it using --lib libraries/....

I needed to add dynamic proxies for the server to run without runtime errors.

Other than that the REPL works (as far as I can see) and effekt -s works too now. The VS code extension itself fails silently so I assume there are still some missing proxies or something. I'm not really sure where to start debugging this.

b-studios · 2023-12-06T15:04:51Z

The missing proxies are often the problem. I remember that there was a tool that would record a trace and then I needed to use as much of the language server as possible...

Edit: https://www.graalvm.org/latest/reference-manual/native-image/metadata/AutomaticMetadataCollection/

b-studios · 2023-12-06T15:05:46Z

Do the missing proxies show in the vscode log?

b-studios · 2023-12-07T16:49:28Z

.github/workflows/deploy.yml

+ run: sudo apt-get install chezscheme
+
+ - name: Run tests and assemble jar file
+ run: sbt clean deploy


@jiribenes just told me MacOS minutes are much more expensive. Maybe we don't want to run the tests on all systems??? :)

marvinborner · 2023-12-07T16:53:11Z

I was now able to debug the server with stacktrace logs. It turns out we do still need reflection files for certain operations (especially lsp4j), so I added missing reflections to the existing config.

With the final config I was not able to produce any reflection errors, but there may still be some. I'm not sure about the general approach of releasing native images, since the reflection fixing does not really seem stable.

Also, saving files with the native build in vscode reports a NullPointerException in kiama.util.Services.formatting$$anonfun$1(Server.scala:572). ~~I'm not sure if it's related to native-image or something else.~~

b-studios · 2023-12-07T17:15:03Z

I'm not sure about the general approach of releasing native images, since the reflection fixing does not really seem stable.

Yeah, that was my feeling last time as well. I would feel better if there was an automated way to query the language server and reproduce scenarios so that we can regenerate reflection traces if necessary (like when updating the LSP4J dependency).

Eventually, reimplementing LSP4J in Scala or replacing it with a Scala implementation that does not use reflection might be the best thing to do but is a LOT of work.

marvinborner · 2023-12-07T17:27:25Z

Maybe writing a general test suite for the LSP server could be a good first step? In my efforts to find reflection bugs I wrote some tests using Haskell's lsp-test. I could start writing something similar (more minimal) in Scala.

b-studios · 2023-12-08T19:55:00Z

Just to make this clear: a test suite for LSP would be great. How would you run it with the graal binary? The tests are not deployed with the binary, no?

marvinborner · 2023-12-08T20:23:09Z

In the case of lsp-test, the tests simply open separate effekt -s sessions and interact with them like any other LSP client -- the tests are 100% external. I thought about implementing the test suite in a similar way (i.e. as a seperate LSP testing program). If we wanted something more performant we could perhaps use the server's debug port.

build.sbt

b-studios · 2023-12-18T10:06:35Z

I was now able to build the binary. However, just invoking effekt -- that is starting the REPL -- crashes with:

java.lang.NoSuchMethodException: jline.UnixTerminal.<init>()

  _____     ______  __  __     _    _
 (_____)   |  ____|/ _|/ _|   | |  | |
   ___     | |__  | |_| |_ ___| | _| |_
  (___)    |  __| |  _|  _/ _ \ |/ / __|
  _____    | |____| | | ||  __/   <| |_
 (_____)   |______|_| |_| \___|_|\_\\__|

 Welcome to the Effekt interpreter (v0.2.1). Enter a top-level definition, or
 an expression to evaluate.

 To print the available commands, enter :help

[ERROR] Failed to construct terminal; falling back to unsupported
java.lang.InstantiationException: jline.UnixTerminal
	at java.base@11.0.17/java.lang.Class.newInstance(DynamicHub.java:571)
	at jline.TerminalFactory.getFlavor(TerminalFactory.java:205)
	at jline.TerminalFactory.create(TerminalFactory.java:96)
	at jline.TerminalFactory.get(TerminalFactory.java:180)
	at jline.TerminalFactory.get(TerminalFactory.java:186)
	at kiama.util.JLineConsole$.terminal(Console.scala:54)
	at kiama.util.JLineConsole$.withTerminal(Console.scala:62)
	at kiama.util.JLineConsole$.usingTerminal(Console.scala:76)
	at kiama.util.JLineConsole$.reader$lzyINIT1(Console.scala:88)
	at kiama.util.JLineConsole$.reader(Console.scala:83)
	at effekt.Repl.usingCommandHistory(Repl.scala:280)
	at effekt.Repl.run(Repl.scala:48)
	at effekt.Driver.run(Driver.scala:39)
	at effekt.Driver.run$(Driver.scala:25)
	at effekt.Server$.run(Server.scala:263)
	at effekt.Server$.run(Server.scala:263)
	at kiama.util.Compiler.driver(Compiler.scala:86)
	at kiama.util.Compiler.driver$(Compiler.scala:22)
	at effekt.Server$.driver(Server.scala:263)
	at kiama.util.Compiler.main(Compiler.scala:50)
	at kiama.util.Compiler.main$(Compiler.scala:22)
	at effekt.Server$.main(Server.scala:263)
	at effekt.Server.main(Server.scala)
Caused by: java.lang.NoSuchMethodException: jline.UnixTerminal.<init>()
	at java.base@11.0.17/java.lang.Class.getConstructor0(DynamicHub.java:3349)
	at java.base@11.0.17/java.lang.Class.newInstance(DynamicHub.java:556)
	... 22 more

b-studios · 2023-12-18T10:16:06Z

The native version reduces the startup significantly (for compiling and running examples/builtins.effekt):

Jar

Time (mean ± σ):      1.179 s ±  0.035 s    [User: 2.660 s, System: 0.115 s]
Range (min … max):    1.136 s …  1.267 s    10 runs

Native

Time (mean ± σ):     353.4 ms ±   1.3 ms    [User: 331.2 ms, System: 16.4 ms]
Range (min … max):   351.5 ms … 355.3 ms    10 runs

Add argument splitting flag for GNU env

47ac4ed

Experiments with scala-native

3dc7bea

marvinborner changed the title ~~Add argument splitting flag for GNU env~~ Create distributable cross-platform packages Dec 4, 2023

marvinborner marked this pull request as draft December 4, 2023 15:22

marvinborner and others added 2 commits December 6, 2023 13:35

Merge branch 'effekt-lang:master' into master

3da5d55

Revert to using sbt-native-image using GraalVM

e8e5e93

Add missing native-image settings

addfc32

marvinborner force-pushed the master branch from 131b15b to addfc32 Compare December 6, 2023 14:14

marvinborner added 2 commits December 6, 2023 15:47

Matrix build workflow POC

5928dc5

Formatting and macos target name fix

c19c98a

Fix missing reflection options

a6aa17d

b-studios reviewed Dec 7, 2023

View reviewed changes

Fix NPE by adding missing reflections

7258a47

b-studios reviewed Dec 18, 2023

View reviewed changes

build.sbt Show resolved Hide resolved

marvinborner mentioned this pull request Dec 18, 2023

Add testing suite for LSP server #351

Open

marvinborner and others added 3 commits February 8, 2024 18:20

Use explicit nativeImage version

43908a8

Add missing reflections so effekt-lang#351 works natively

743b3f6

Merge branch 'effekt-lang:master' into master

8bc6674

marvinborner mentioned this pull request Mar 5, 2024

Difficulties with the Effekt installation on Ubuntu #407

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create distributable cross-platform packages #329

Create distributable cross-platform packages #329

marvinborner commented Dec 3, 2023

b-studios commented Dec 3, 2023 •

edited

Loading

marvinborner commented Dec 3, 2023 •

edited

Loading

jiribenes commented Dec 3, 2023 •

edited

Loading

b-studios commented Dec 3, 2023

marvinborner commented Dec 4, 2023

b-studios commented Dec 4, 2023

marvinborner commented Dec 6, 2023 •

edited

Loading

b-studios commented Dec 6, 2023 •

edited

Loading

marvinborner commented Dec 6, 2023

b-studios commented Dec 6, 2023 •

edited

Loading

b-studios commented Dec 6, 2023

b-studios Dec 7, 2023

marvinborner commented Dec 7, 2023 •

edited

Loading

b-studios commented Dec 7, 2023

marvinborner commented Dec 7, 2023

b-studios commented Dec 8, 2023

marvinborner commented Dec 8, 2023 •

edited

Loading

b-studios commented Dec 18, 2023

b-studios commented Dec 18, 2023

Create distributable cross-platform packages #329

Are you sure you want to change the base?

Create distributable cross-platform packages #329

Conversation

marvinborner commented Dec 3, 2023

b-studios commented Dec 3, 2023 • edited Loading

marvinborner commented Dec 3, 2023 • edited Loading

jiribenes commented Dec 3, 2023 • edited Loading

b-studios commented Dec 3, 2023

marvinborner commented Dec 4, 2023

b-studios commented Dec 4, 2023

marvinborner commented Dec 6, 2023 • edited Loading

b-studios commented Dec 6, 2023 • edited Loading

marvinborner commented Dec 6, 2023

b-studios commented Dec 6, 2023 • edited Loading

b-studios commented Dec 6, 2023

b-studios Dec 7, 2023

Choose a reason for hiding this comment

marvinborner commented Dec 7, 2023 • edited Loading

b-studios commented Dec 7, 2023

marvinborner commented Dec 7, 2023

b-studios commented Dec 8, 2023

marvinborner commented Dec 8, 2023 • edited Loading

b-studios commented Dec 18, 2023

b-studios commented Dec 18, 2023

b-studios commented Dec 3, 2023 •

edited

Loading

marvinborner commented Dec 3, 2023 •

edited

Loading

jiribenes commented Dec 3, 2023 •

edited

Loading

marvinborner commented Dec 6, 2023 •

edited

Loading

b-studios commented Dec 6, 2023 •

edited

Loading

b-studios commented Dec 6, 2023 •

edited

Loading

marvinborner commented Dec 7, 2023 •

edited

Loading

marvinborner commented Dec 8, 2023 •

edited

Loading