DX for authoring account and note scripts with cargo-miden

[greenhat]

Creating a new project

To create a new account project:

cargo miden new my-account

with --account implied, or specifying it explicitly:

cargo miden new --account my-account

To create a new note script project:

cargo miden new --note my-note

WIT bindings and rust-analyzer

rust-analyzer depends on cargo check to check for errors. To ensure that rust-analyzer can discover the latest bindings (generated from the WIT file), rust-analyzer must be configured to use cargo miden check as the check command. The cargo miden check will re-generate the bindings if the WIT file is changed.

By default, cargo miden new will configure Visual Studio Code to use cargo miden check by creating a .vscode/settings.json file for you.

Specifying the project's Miden package dependencies

The project dependencies on Miden package(s) are specified in the Cargo.toml file of the project in a separate table package.metadata.miden. For example, to specify the dependency on the specific account package in the note script project:

[package.metadata.miden]
# Miden package id
package = "miden:note-script"

[package.metadata.miden.dependencies]
"miden:basic-wallet" = { path = "SOME_PATH" }

Where SOME_PATH is either a path to the Miden package file or a Rust project from which the Miden package can be built.

For the MVP we only support local dependencies (path field) with package repository support coming later.

Compiling the Miden package

To compile the Miden package with dev(debug) profile:

cargo miden build

Which builds the Miden package and puts it in the target/miden/debug directory.

To compile the Miden package with release profile:

cargo miden build --release

Which builds the Miden package and puts it in the target/miden/release directory.

Running/debugging the Miden package

The account does not have an entry point, so it is not possible to run it directly. We could specify an entry point and inputs via the command line, but I believe such use cases would be easier for the MVP to cover with our testing infrastructure.
The note script can be run directly, but it requires an account, in most cases pre-populated with some state (assets, etc.) which is cumbersome to do via the command line and would be easier to cover with our testing infrastructure.

Testing the Miden package

Since the account API is called only from the note script (for now at least), we can focus on providing enough glue (test infrastructure) to let note scripts interact with the account and make assertions of its state.

Building on the ideas of the Smart contracts testing discussion and miden-base testing infrastructure, we can simulate the rollup and run the compiled Miden packages for account and note scripts on the VM and assert the state of the account after the execution.

The simplest test for the basic wallet account would look like this:

#[test]
fn test_basic_wallet_alice_sends_bob() {
    // The rollup simulator
    let rollup = MockRollup::new();
    let alice_account_package = compile("path/to/alice_account_rust_crate");
    let alice = rollup.crate_account(alice_account_package);
    let bob_account_package = compile("path/to/bob_account_rust_crate");
    let bob = rollup.create_account(bob_account_package);
    // Mints and deposits some coins to the account
    alice.deposit(Asset::new("ACoin", 100));
    let note1_script = compile("path/to/note1_rust_crate");
    // Execute a tx which creates a note with a given note script as a recipient
    let note1 = alice.create_note(note1_script, [bob.id()], Asset::new("ACoin", 90));
    // Directly create a tx with the given input notes 
    let bob_tx = Transaction::load_account(bob)
        .input_note(note1, [bob.id()])
        .execute();
    assert_eq!(alice.assets_quantity("ACoin"), 10);
    assert_eq!(bob.assets_quantity("ACoin"), 90);
}

Where compile compiles the Miden package (equivalent to cargo miden build) and returns it wrapped in some test harness.

This way we only simulate the rollup parts and running the Miden packages under test on the VM.

The tests are regular Rust integration tests and can be run with cargo miden test which will run cargo test under the hood.

To debug the account and note scripts in the MVP we can control the debugger from the test code. For example, to stop on entering the receive_asset function of the basic wallet account put in the test code:

rollup.debug.break_on_entry("basic_wallet:receive_asset");

Which will stop the execution on the first call to the receive_asset function of the basic_wallet and open the debugger TUI.
Later, we can utilize the DAP protocol and expose the debugger to the IDE/editor and utilize the debugging capabilities of the IDE/editor.

Deploying the Miden package

To deploy the Miden package to a Miden node use the deploy command:

cargo miden deploy [OPTIONS] 

@bitwalker @bobbinth

[bobbinth]

Great writeup! Thank you! A few preliminary questions/comments:

To be able to run cargo miden, we need to install cargo-miden crate, right? e.g., via cargo install cargo-miden.

Not for now, but maybe later, do we envision projects that would have accounts and notes together (in different crates)? In the current setup, do we put anything into the Cargo.toml file to specify that a given crate is for a note vs. an account (kind of like bin vs lib)?

Are miden-stdlib and miden-lib dependencies available by default?

Also, regarding dependencies: I think it makes sense for accounts to depend on other accounts, and for notes to depend on accounts, but it probably doesn't make sense for accounts to depend on notes, or for notes to depend on notes. Is this something we'd want to enforce?

In general, how do we specify library dependencies (i.e., not note or account dependencies)?

What exactly would account package contain? I've written up some thoughts in 0xPolygonMiden/miden-base#550 (comment) - but there are more questions than answer there. I think the main thing is figuring out how to describe the "shape" or the storage and how it should be initialized on deployment.

In general, I'm wondering if deployment should be handled separately. That is, the job of cargo miden would be to create an account or a note package (and in the future to upload it to some registry), but the actual deployment would be done separately (e.g., by miden-client). The reason for this is that deploying a contract requires quite a bit of other stuff (e.g., we may need to provide a public key from a specific wallet).

[greenhat]

To be able to run cargo miden, we need to install cargo-miden crate, right? e.g., via cargo install cargo-miden.

Yes, that's right.

Not for now, but maybe later, do we envision projects that would have accounts and notes together (in different crates)?

Do you mean in the same workspace? It should work now. I mean, I don't see any reason why it wouldn't work.
We should add a check that "call"-able crates (like account in a note script) are not used as crate dependencies but as Miden packages. Other than that, it should work.

In the current setup, do we put anything into the Cargo.toml file to specify that a given crate is for a note vs. an account (kind of like bin vs lib)?

Good catch! No, but we should. It could be package.metadata.miden.kind with the value bin or lib. I wonder, since we also have a tx scripts, do we need to distinguish them and note scripts? If yes, lib/bin is not enough and we need a third one.

Are miden-stdlib and miden-lib dependencies available by default?

Yes, they are set as crate dependencies in the Cargo.toml file in new project templates.

Also, regarding dependencies: I think it makes sense for accounts to depend on other accounts, and for notes to depend on accounts, but it probably doesn't make sense for accounts to depend on notes, or for notes to depend on notes. Is this something we'd want to enforce?

So far, I see the WIT interface for a script note to have a single fixed-name (note-script) function without arguments that returns nothing. So it's not that useful to have a dependency on a note. We can catch those cases and report a warning to prevent accidentally depending on a note.

In general, how do we specify library dependencies (i.e., not note or account dependencies)?

As regular crate dependencies. Anything, that should be exec-ed and not call-ed, should be directly linked as a crate dependency.

What exactly would account package contain?

We have #129 with initial package documentation (rendered).

I've written up some thoughts in 0xPolygonMiden/miden-base#550 (comment) - but there are more questions than answer there. I think the main thing is figuring out how to describe the "shape" or the storage and how it should be initialized on deployment.

For a semi-high-level API, I'm thinking along the lines:

#[account_storage]
struct MyAccountStorage {

    #[storage_slot(0)]  
    foo: Word,

    #[storage_slot(1)]  
    pub_key: Vec<Word>,

    #[storage_slot(2)]  
    misc: Word,
}

The attribute macros generate getters and setters (MyAccountStorage::foo().get(), .set(value: Word)) for the fields which call get_item and set_item on the account storage.

For a more high-level API and stronger typing, something like this might work:

#[account_storage]
struct MyAccountStorage {

    #[storage_slot(0)]  
    foo: MyFooType,

    #[storage_slot(1)]  
    pub_key: RpoFalcon512Key,

    #[storage_slot(2)]  
    misc: MyMiscType,
}

Where user types need to implement some Encode and Decode traits for serialization and deserialization.

We can probably utilize the link_section attribute to store the binary serialized description of the account storage in the Wasm binary. So that it would end up in the Miden package.

The backup plan is to define the account storage structure in the Cargo.toml or any other file and make cargo miden check generate Rust bindings for it.

Initial account storage value initialization needed for a deployment could be defined in the separate crate, compiled into a Wasm component and included in the Miden package.

In general, I'm wondering if deployment should be handled separately. That is, the job of cargo miden would be to create an account or a note package (and in the future to upload it to some registry), but the actual deployment would be done separately (e.g., by miden-client). The reason for this is that deploying a contract requires quite a bit of other stuff (e.g., we may need to provide a public key from a specific wallet).

Sure. I included it here without giving too much thought. Deployment is quite complex process, and it's a good idea to handle it separately.

[bobbinth]

I wonder, since we also have a tx scripts, do we need to distinguish them and note scripts? If yes, lib/bin is not enough and we need a third one.

I think we might have to. So, maybe the 3 types we need are: account, note, script.

For a semi-high-level API, I'm thinking along the lines:

#[account_storage]
struct MyAccountStorage {

    #[storage_slot(0)]  
    foo: Word,

    #[storage_slot(1)]  
    pub_key: Vec<Word>,

    #[storage_slot(2)]  
    misc: Word,
}

This could work, but a few comments:

• Could storage slot indexes be inferred? (i.e., do we really need storage_slot attributes?)
• I don't think we should allow using regular vectors for storage. We should probably provide StorageMap (and in the future StorageVec) structs. Only these + Word could be used as types for account storage.
  • We could relax this in the future to allow other types and then somehow automatically compute packing/layout.

There is another way to think about account packages: these could be something that don't describe an account fully - but describe a "component" of an account (and here "component" is different from WASM components). Let's take our basic wallet account for example. It actually consists of 2 components: "basic wallet" (the send/receive asset procedures) and the authentication component (we currently use RpoFalcon signature).

So, maybe these are two separate packages which we combine at deployment time to build an account. If we do it this way, then storage offsets get assigned at deployment time - not at "compile" time. I haven't thought this through completely - but this could have a number of advantages. (also cc @phklive as this may impact how we instantiate accounts in miden-base).

[greenhat]

I wonder, since we also have a tx scripts, do we need to distinguish them and note scripts? If yes, lib/bin is not enough and we need a third one.

I think we might have to. So, maybe the 3 types we need are: account, note, script.

Sounds good.

For a semi-high-level API, I'm thinking along the lines:

#[account_storage]
struct MyAccountStorage {

    #[storage_slot(0)]  
    foo: Word,

    #[storage_slot(1)]  
    pub_key: Vec<Word>,

    #[storage_slot(2)]  
    misc: Word,
}

This could work, but a few comments:

• Could storage slot indexes be inferred? (i.e., do we really need storage_slot attributes?)

Probably yes, by relaying on the order of the fields in the struct. I see two drawbacks with it:

1. Moving fields in the struct will change their slot indices, which most likely will surprise the developer when they discover it later on (when?).
2. By implicitly assigning slot indices, we're hiding the important inner workings of the account storage system from the developer.

• I don't think we should allow using regular vectors for storage. We should probably provide StorageMap (and in the future StorageVec) structs. Only these + Word could be used as types for account storage.

  • We could relax this in the future to allow other types and then somehow automatically compute packing/layout.

Yes, you're right. Generic Vec will not be enough in encode storage type invariants we want to enforce.

There is another way to think about account packages: these could be something that don't describe an account fully - but describe a "component" of an account (and here "component" is different from WASM components). Let's take our basic wallet account for example. It actually consists of 2 components: "basic wallet" (the send/receive asset procedures) and the authentication component (we currently use RpoFalcon signature).

I've gone through the miden_base. Do you mean auth_tx_rpo_falcon512 procedure that is exported from the account and called in the tx script? We can probably link it on deployment from MASM. How do we call it in the tx script, though? I mean, when the tx is created, the developer has to provide the tx script.

[bobbinth]

Probably yes, by relaying on the order of the fields in the struct. I see two drawbacks with it:

1. Moving fields in the struct will change their slot indices, which most likely will surprise the developer when they discover it later on (when?).
2. By implicitly assigning slot indices, we're hiding the important inner workings of the account storage system from the developer.

Agreed. I think it makes sense to start with defining the indexes explicitly and then we can figure out later if there is a good way to make them implicit.

I've gone through the miden_base. Do you mean auth_tx_rpo_falcon512 procedure that is exported from the account and called in the tx script? We can probably link it on deployment from MASM. How do we call it in the tx script, though? I mean, when the tx is created, the developer has to provide the tx script.

Yes, in most cases a transaction will be accompanied by a tx script which would be responsible for calling the authentication procedure for the account (the exception to this would be accounts which can be updated w/o authentication). This is handled by the client though. That is, the client knows that RpoFalcon512 is the authentication procedure for the account and so it automatically creates a tx script based on that.

Coming back to "account components", I'm thinking AccountCode in miden-base could be instantiated from account components rather than a single library (as it is now). For example, something like:

impl AccountCode {
    pub fn new(components: &[AccountComponent], storage_offset: u8) -> Result<Self, AccountError> {
        /// combine all component libraries into a single library, apply storage offsets etc.
    }
}

/// Minimal version - probably should have other fields.
struct AccountComponent {
    code: Library,
    storage: Vec<StorageSlotType>,
}

The AccountComponenet struct would be something that can be instantiated from the account component package produced by the compiler.

In the context of the basic wallet account, there would be 2 components:

1. The basic wallet interface with send/receive asset procedures. This interface does not use storage and so storage vector of AccountComponent would be empty.
2. The RpoFalcon512 authenticator. This interface requires a single storage slot with the public key. So, the storage vector would look like vec![StorageSlotType::Value].

I think this could work quite well. The main remaining thing is to figure out how to handle/package "storage initializers".

[greenhat]

Looks doable. I hope we can "smuggle" the account storage description in link_section attribute to store the binary serialized description of the account storage in the Wasm binary. If not, then we pass it out of band. So that it would end up in the Miden package and loaded into AccountComponent.