fix: add tx kernel library with stubs and link it on -l base

[greenhat]

This PR adds miden tx kernel stubs and links them via -l base compiler option

[greenhat]

@bitwalker I'm having second thoughts about the naming. Maybe it's better to be explicit that it's not the real tx kernel, only stubs? miden-tx-kernel-stubs, etc.

[bitwalker]

My understanding was that we were going to name wrapper crates like this something to the effect of miden-<name>-sys.
where <name> is something like stdlib, tx_kernel, etc.

While this crate doesn't contain bindings yet, I would expect that whatever crate does contain bindings (or is expected to eventually), would also be responsible for producing the Library (and embedding it in the crate for distribution).

The Miden standard library is a bit weird at the moment, as it is currently maintained in miden-vm, and the miden-stdlib crate does not contain bindings, instead we have -sys crates in this repo which handle that, but the Library is embedded in miden-stdlib. But I wouldn't pay much mind to that - instead, we should define miden-base-sys, and have that define a Library containing the code from miden-base (or our stubs, as the case may be for the moment). We would then link it using -l base.

I'm not sure we need to have separate crates for each subcomponent of miden-base, since it is basically all-or-nothing, but I'm open to reconsidering that aspect.

[greenhat]

My understanding was that we were going to name wrapper crates like this something to the effect of miden-<name>-sys. where <name> is something like stdlib, tx_kernel, etc.

Yes, but I have not connected the dots and started a new crate. 🤦

While this crate doesn't contain bindings yet, I would expect that whatever crate does contain bindings (or is expected to eventually), would also be responsible for producing the Library (and embedding it in the crate for distribution).

Makes sense. Does it mean that we no longer need to separately specify digests for each function in the bindings? Like in

compiler/sdk/tx-kernel-sys/src/externs.rs

Lines 6 to 8 in ab2b725

	extern "C" {
	#[link_name = "get_id<0x0000000000000000000000000000000000000000000000000000000000000000>"]
	pub fn extern_account_get_id() -> AccountId;

The Miden standard library is a bit weird at the moment, as it is currently maintained in miden-vm, and the miden-stdlib crate does not contain bindings, instead we have -sys crates in this repo which handle that, but the Library is embedded in miden-stdlib. But I wouldn't pay much mind to that - instead, we should define miden-base-sys, and have that define a Library containing the code from miden-base (or our stubs, as the case may be for the moment). We would then link it using -l base.

I'm not sure we need to have separate crates for each subcomponent of miden-base, since it is basically all-or-nothing, but I'm open to reconsidering that aspect.

I agree.

[bitwalker]

Makes sense. Does it mean that we no longer need to separately specify digests for each function in the bindings?

My thinking was that the build script for the crate would compile the MAST, write that to OUT_DIR (and then store it in the crate sources using include_bytes!), as well as generate a Rust source file containing the extern function definitions for the procedures in the MAST, using the digests that we can get from the Library struct by visiting each module and procedure, paired with signature metadata that we get initially from something we maintain by hand (maybe JSON? doesn't matter, whatever is most convenient, could also add a dependency on midenc-hir-type and serialize types to/from JSON), but when I am able to land the changes that add type signatures to Miden Assembly, we would derive the signature metadata from that. So essentially we are able to automate the generation of bindings for the most part for -sys crates.

The aspect that would be hand-maintained, would be the high-level Rust crate that depends on the -sys crate - these would provide more native abstractions for the underlying raw procedure bindings.

This gets us the ability to load MAST from a Rust crate in the compiler (or in the test suite, etc.), like we do with miden-stdlib, the ability to generate the bindings is at nice benefit as well. We may want to use a separate process for managing the compilation of the MAST, and simply read the existing MAST to generate the bindings, but for now this will simplify some things.

[bitwalker]

A few things:

1. We should not use miden as the name for any Library, as it is far too general and ambiguous
2. If we are going to have separate crates for each miden-base subcomponent (such as the TX kernel), we should be more precise with the names, e.g. -l miden_tx_kernel (which should then be the file stem used when writing it to disk). I add the miden_ prefix here because if we have a number of such libraries, they are less "special", so should be namespaced to avoid conflicts
3. I would prefer to group miden-base libraries together, and link them using -l base. This would be a "special" library like std, which we handle in the compiler directly.
4. Rather than specifying -l std -l base, we should infer those options in the compiler when --target rollup is specified (since it is implied that those libraries are dependencies). Here, in cargo-miden, we should distinguish between building a crate for the rollup, and building a crate that just targets the base VM, and pass --target to get the default set of libraries for those targets.

I'll add target-derived default libraries today, so you can assume the default target adds -l std, and the rollup target adds both -l std and -l base; but I'll leave it up to you to add handling for -l base in the LinkLibrary type of midenc-session like we do for -l std

[greenhat]

I don't think we need one crate for each miden-base subcomponent. I agree that miden is too generic.

[bitwalker]

A few other suggested tweaks, primarily just to ensure we have usable debug info

[bitwalker]

I think you can simply make the masl-lib parts of this use #[cfg(not(target_family = "wasm"))], rather than a Cargo feature. This keeps it out of the Wasm builds that would cause this to end up in code processed by the compiler, but otherwise keeps all of the code together.

In the future we may need to add a feature that allows those items to exist in a Wasm build as well (for building the compiler to run in the browser), but we're a way away from that point yet.

[bitwalker]

Or vice versa, make the bindings conditional on #[cfg(target_family = "wasm")] - that might make more sense, since the bindings only are needed when targeting Miden.

[greenhat]

@bitwalker This PR is ready. The only test that fails is get_inputs.

[greenhat]

@bitwalker I've rebased the PR and it's ready.