Introduce --no-import-callback CLI option

[nikola-matic]

Fixes: #14484

• --no-import-calback
• Tests
• Docs

[nikola-matic]

This I find somewhat suspicious, and kinda stumbled upon it by accident. Reading sources via URLs in the standard compiler requires having the file reader callback set, which means that --no-import-callback essentially disabled this feature (i.e. clashes with --allowed-paths. The question now is - firstly, is this OK, and if yes, should I think disallow usage of --no-import-callback and --allowed-paths together?

[cameel]

I see you're slowly discovering all the intricacies of the callback and VFS the hard way, like I did back when I dug into this topic last time :) I think it would be better for you not to retread that path, and just read through the docs I wrote back then: Import Path Resolution. It's a dull topic, but you should know it, because there are a lot of quirks related to file loading and imports that start to make sense when you know how it works as a whole.

The urls thing is mentioned in Initial Content of the Virtual Filesystem. The name itself shows that it's more than just a filesystem loader. Historically, the idea would be that imports are not necessarily filesystem paths. They're just some unique identifiers that your callback can interpret and use to load the source from. For example Remix to this day allows you to import files directly from Github - which is achieved via a custom callback. urls is an extension to that - it lets you load a file from an external source but without embedding that knowledge into the import. Your import refers to a generic source unit name and in Standard JSON you provide a list of URLs where the content matching that source unit name can be found. The first one that can be loaded by your callback is used. For example, that way you could have input that will use OZ from your local machine when available and if not, it will try to load it from github. Or you could keep a version embedded in the URL to make sure the file does not change under you, but not have to repeat it every time you import the file in you sources.

which means that --no-import-callback essentially disabled this feature

Yes, that was my intention here. The main use case of --no-import-callback is to have the compiler treat Standard JSON as self-contained, without reaching into the filesystem. And the current behavior fits that use case very well.

clashes with --allowed-paths

I can't see how they would clash. Allowed paths are a feature of the default callback. If you disable the callback, they have no effect.

should I think disallow usage of --no-import-callback and --allowed-paths together

Yeah, it makes sense to disallow using them together, but more due to the fact that you would never need to do that and allowing them both may give a different impression.

For context: there are 3 options that end up being stored in FileReader: --allow-paths, --include-path and --base-path. Of those --allow-paths is the only one relevant to the callback but not to processing files given on the CLI. The latter two are relevant to both - they are needed to translate paths to source unit names, i.e. the normalized names that we use in CompilerStack and everywhere inside the compiler to refer to files.

[cameel]

Here's some comments. I also pushed a few smaller tweaks as a fixup.

[cameel]

I see you're slowly discovering all the intricacies of the callback and VFS the hard way, like I did back when I dug into this topic last time :) I think it would be better for you not to retread that path, and just read through the docs I wrote back then: Import Path Resolution. It's a dull topic, but you should know it, because there are a lot of quirks related to file loading and imports that start to make sense when you know how it works as a whole.

The urls thing is mentioned in Initial Content of the Virtual Filesystem. The name itself shows that it's more than just a filesystem loader. Historically, the idea would be that imports are not necessarily filesystem paths. They're just some unique identifiers that your callback can interpret and use to load the source from. For example Remix to this day allows you to import files directly from Github - which is achieved via a custom callback. urls is an extension to that - it lets you load a file from an external source but without embedding that knowledge into the import. Your import refers to a generic source unit name and in Standard JSON you provide a list of URLs where the content matching that source unit name can be found. The first one that can be loaded by your callback is used. For example, that way you could have input that will use OZ from your local machine when available and if not, it will try to load it from github. Or you could keep a version embedded in the URL to make sure the file does not change under you, but not have to repeat it every time you import the file in you sources.

which means that --no-import-callback essentially disabled this feature

Yes, that was my intention here. The main use case of --no-import-callback is to have the compiler treat Standard JSON as self-contained, without reaching into the filesystem. And the current behavior fits that use case very well.

clashes with --allowed-paths

I can't see how they would clash. Allowed paths are a feature of the default callback. If you disable the callback, they have no effect.

should I think disallow usage of --no-import-callback and --allowed-paths together

Yeah, it makes sense to disallow using them together, but more due to the fact that you would never need to do that and allowing them both may give a different impression.

For context: there are 3 options that end up being stored in FileReader: --allow-paths, --include-path and --base-path. Of those --allow-paths is the only one relevant to the callback but not to processing files given on the CLI. The latter two are relevant to both - they are needed to translate paths to source unit names, i.e. the normalized names that we use in CompilerStack and everywhere inside the compiler to refer to files.

[cameel]

I have some minor nitpicks, but other than that everything is in order and I tentatively approve.

Please remember to either squash before merging or fix your commits - the current description is a lie :P