Add # elixir-mode: to .exs files to enable Mix.install/imports in same file

[thmsmlr]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.0.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.17.2 (compiled with Erlang/OTP 27)

Operating system

MacOS

Current behavior

Currently, when writing Elixir Scripts, if you Mix.install a dependency and try to import it in the same file at the top-level it doesn't work (see issues).

This happens because the default execution mode for elixir scripts is to compile the entire file and run. Whereas if the script was run expression at a time, like if you were to run Mix.install() then import in an IEX shell, you'd get the desired behavior.

This rears its head in other situations too, for example, you can't define a struct and use it in top-level in the same elixir script either.

defmodule Person do
  defstruct [:first_name, :last_name]
end

person = %Person{first_name: "John", last_name: "Doe"}

IO.inspect(person, label: "person")

$ elixir demo.exs
    error: cannot access struct Person, the struct was not yet defined or the struct is being accessed in the same context that defines it
    │
  6 │ person = %Person{first_name: "John", last_name: "Doe"}
    │          ^
    │
    └─ /Users/thomas/code/scrapers/demo.exs:6:10

** (CompileError) /Users/thomas/code/scrapers/demo.exs: cannot compile file (errors have been logged)

This feels like something you'd really want to be able to do in a scripting environment, especially now with Mix.install and projects like Livescript, or Livebook when exporting to .exs

What's interesting is that this has already been addressed in IEX when using a ~/.iex.exs file to solve exactly this (See commit). Notably, however it is only for ~/.iex.exs files and not for files passed into IEX as a CLI argument.

$ iex demo.exs 
Erlang/OTP 27 [erts-15.0.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

    error: cannot access struct Person, the struct was not yet defined or the struct is being accessed in the same context that defines it
    │
  6 │ person = %Person{first_name: "John", last_name: "Doe"}
    │          ^
    │
    └─ demo.exs:6:10

** (CompileError) demo.exs: cannot compile file (errors have been logged)

Expected behavior

Based on a conversation with @josevalim on X. A solution could be to add a # elixir-mode: pragma to elixir scripts that allows the user to define how the script should be executed.

1. compile (the default)
2. eval (like Code.eval_file)
3. expr-eval (new one but behaves like IEx)

When scripts are put into # elixir-mode: expr-eval mode, the script will be evaluated expression by expressions allowing for things like Mix.install & import to be used within the same script. As well as using structs top-level that were defined in the same file.

[Eiji7]

Forgive me, but we can define the mode wherever we want (comment, CLI argument or so), right? In that case instead of # elixir-mode: expr-eval I would like to suggest passing such option to elixir.

$ elixir example.exs --mode expr-eval

Since we are doing this for Elixir I believe this is much better solution, because we can put such thing at the top:

#!/usr/bin/env elixir --mode expr-eval
# ^^^ Let shell handle above ^^^

defmodule Person do
  defstruct [:first_name, :last_name]
end

person = %Person{first_name: "John", last_name: "Doe"}
IO.inspect(person, label: "person")

so we can add executable permission using chmod +x example.exs and execute such script as any other ./example.exs.

If what I wrote is possible then adding "special comment" does not makes much sense as we (or rather the shell) already supports one special comment which we can use.

What do you think about it?

[thmsmlr]

I don't have a strong opinion, for my usecase that would work just fine.

However for completeness, I can see one downside. Making it only a CLI flag in the shebang is that one might write a script that expects --mode expr-eval and someone else goes to execute it not knowing that, and runs it the normal way that you would expect elixir example.exs and the script would fail in a confusing way.

In my mind the evaluation mode is a concern of the script being run, not that of who is running it.

Now I don't know the internals of Elixir well enough to say for certain, but there also may be some weird interplay with import_file() using the shebang method since it presumably doesn't read or consider the shebang at all (rightfully).

Feels to me the special comment is the more robust solution and more forward looking, but as I said, I personally chmod +x all my elixir scripts so it wouldn't hit me.

	./example.exs	elixir example.exs	import_file()?
#!/usr/bin/env elixir --mode expr-eval	✓	✗	?
# elixir-mode: expr-eval	✓	✓	✓

Either way, works for me though

[Eiji7]

I see the point, but running a executable script with elixir is edge case and there are few ways to workaround it:

1. Improve error message in mentioned use cases by adding information about said option
2. Make this option default for only elixir executable (i.e. not iex, mix, executable on prod server or whatever else)
3. Other support in elixir executable (warn by default, catch error/return and warn or parse hashbang)

Support hashbang for example could be very simple as all we have to do is support string #!/usr/bin/env elixir and take rest as arguments. We can merge, override all or ask for a decision - there are plenty of options. For sure there are other hashbangs possible, but we don't have to handle them. We can simply warn about unsupported hashbang which would be same for a custom Elixir path as for a python script (another edge case).

some language interpreters that do not use the hash mark to begin comments still may ignore the shebang line in recognition of its purpose

Source: https://en.wikipedia.org/wiki/Shebang_(Unix)

However I would still prefer 1st option, because:

1. It's even easier than supporting hashbang
2. Clear message would stop some people from creating a duplicated topic on forum
3. It would help developers right away even if they don't know yet about such option

As above a comment way looks more "hidden" - something that people may remove not understanding what it do, because "it's just a comment". It would make people search for the problem without giving any hint.

[wojtekmach]

I’m very much looking forward to this!

I keep thinking about one thing though, I wonder if instead of having a magical comment (and setting a precedent for it), could we have something like:

Code.mode(:eval_expr)

which would look like regular code but it would be totally hardcoded by tokenizer/parser (just like a magic comment would) i.e. can only be the very first line (optionally after unix shebang).

If that would be possible, I think the biggest benefit would be that we could document it.

[thmsmlr]

I'm not particularly convinced by those solutions. In the case of error messages you'd have to keep a list of growing usecases to catch and detect are a result of the wrong evaluation mode. I don't think we should switch any defaults either, that may have wide spread consequences. I also don't think parsing the hashbang is a good idea, feels like it's just hijacking something we don't control.

Magic comments are pretty common way to solve this problem across languages. C has #pragma, Python has # -*- coding: utf-8 -*-, Rust has #![allow(unused_variables)], Ruby has # frozen_string_literal: true.

Spiritually they're very similar to Elixir's Code.compiler_options(), but this would be runtime variant? Perhaps there's a non-magic comment solution in adding a Code.eval_options()?

One thing I keep coming back to is that the eval mode, to me, feels like a concern of the script author, not the script runner. Therefore, however it is articulated in the script (shebang, magic comment, Code.eval_options(...)), it should be inside the script and respected no matter how it runs, whether it's through elixir, iex, or Code.eval_file()

But ya, once again I want to make clear, even if folks disagree with that perspective, it's not a show stopper for me. A CLI flag (scriptable via a shebang) technically solves my personal problems.

[Eiji7]

Code.mode(:eval_expr)

which would look like regular code but it would be totally hardcoded by tokenizer/parser (just like a magic comment would) i.e. can only be the very first line (optionally after unix shebang).

At first I was not sure what to think about it, but that's actually very interesting idea! Please pay attention that it would not be "just" documented, but it would also be supported out-of-box by any editor/plugin that shows documentation. This is important especially because even if editor/plugin would parse hashbang it would not recognise every option passed to every possible executable and because of that I like this solution the most. ❤️

[josevalim]

Support hashbang for example could be very simple as all we have to do is support string #!/usr/bin/env elixir and take rest as arguments.

Unfortunately, hashbangs don't work on Windows, so we have to rule it out as an option.

Error messages are also trickly because we never have the whole context. It could also happen that the script author wrote a typo, Acount instead of Account, and we will now be telling them "hey, perhaps you want a flag?". Overall I agree this is a responsibility of the script owner.

I keep thinking about one thing though, I wonder if instead of having a magical comment (and setting a precedent for it), could we have something like:

Code.mode(:eval_expr)

I think this could be a good direction, but it should not be a special module+function hardcoded in the compiler. The only times we hardcode things in the compiler are for optimizations, not for semantics. Furthermore, we need to be clear that whatever this is, it must only happen at the top of the file, before any other expression. A similar solution is to introduce a special form called pragma:

pragma file: :compile 

However, solutions in this style have a downside. The reason why pragmas are typically magic comments is because pragmas must be parsed before the contents of the file are parsed. This is not strictly the case for this feature, this feature requires configuring compilation/evaluation after the code is parsed, but there is a chance that we will need an actual pragma in the future, and this approach will either close the door on that or force us to have two somewhat overlapping solutions. For example, we could use pragmas/magic-comments to opt-in to some Unicode features.

[josevalim]

If the concern is documentation, which is definitely a good concern, I am sure we could teach editors to display something. We could even choose a more specific syntax so it is easier to spot

# @pragma compile
# @pragma eval-file
# @pragma eval-expr

or

# elixir: compile
# elixir: eval-file
# elixir: eval-expr

[greven]

If we are going the pragma route, I think the syntax '# @pragma: eval-expr' makes more sense to me. The one with the elixir word would only make sense if we could colide with other pragmas? But that can't happen in an .exs file.

[josevalim]

@thmsmlr just so you know, in Livebook we are telling users to run it as iex --dot-path path/to/script.exs. That will evaluate line by line and give you an environment with all variables at the end. Perhaps you want to do that as well.

[smaximov]

I keep thinking about one thing though, I wonder if instead of having a magical comment (and setting a precedent for it), could we have something like:

Code.mode(:eval_expr)

which would look like regular code but it would be totally hardcoded by tokenizer/parser (just like a magic comment would) i.e. can only be the very first line (optionally after unix shebang).

Hardcoding seems like an interesting solution, but as I understand it would break if we complicate the example a little:

module = Code
module.mode(:eval_expr)

It's not like there's a compelling reason for a user to do such shenanigans, but I don't like an idea of introducing an obscure rule under which a seemingly regular code behaves in an unexpected way. If that is the case, my vote is for a magical comment.