Miden Packaging

[bitwalker]

Here are some notes I sketched up around packaging that we can use as a starting point for discussion. I'll try to come back and expand/update this as our plans evolve:

Packaging

• Contents
  • Manifest (contains all metadata describing the contents and their locations in the package)
  • WIT descriptor (needed to provide metadata about the component defined by the package)
  • MAST (Or if the MAST is stored elsewhere, the MAST root)
  • MASM? (Could be needed in conjunction with debug info)
  • Debug Info (optional)
  • Docs (optional, would be useful for providing something like docs.rs for packages)
• Format
  • Presumably binary, but would be nice to have a corresponding directory layout that can be used
    for staging a package, as well as exploring the contents in the shell natively
  • A format that is compressible and seekable in binary form would be nice. Tar has some useful
    properties here, as it can be extended with vendor-specific extensions via PAX without breaking compatibility
    with tools that understand the tar format, and is easily paired with different compression algorithms
    (e.g. gzip/zstd)
  • Alternatively, we could just zip up the directory layout from above
  • File extension? Some options:
    • .mbpf or .mbp (Miden Binary Package Format, no known uses of these that I'm aware of)
    • .mc or .mcf or .mcp (Miden Component Package/Format)
    • .mastp (MAST package)
    • .masmp (Miden Assembly Package, bit of a misnomer)
• Manifest
  • Manifest version/magic
  • Sizes and offsets of each piece of metadata in the binary blob (for seeking)
  • Package name/version + optional metadata like author/description/etc.
  • Minimum required Miden VM version?
  • MAST root of the entire program/library
  • MAST roots of all exported functions, to be used during compilation
  • What optional components are present (docs, debug info) and their location
• Publishing Packages/Dependency Management
  • Do we host a registry somewhere?
  • Likely would base it on warg, but using our package format
• Versioning
  • It would be ideal to provide both hash and semantic versioning options,
    where a semantic version is resolved and then the hashes of the resolved
    package are used under the covers
  • We'll want a lockfile once a dependency has been resolved to a specific hash
  • We will want to prevent publishing different versions of the same package
    if they contain different hashes for any of the exports, but we could
    allow changes to a package that are superficial in nature (e.g. changing
    documentation, if part of a package)

[bitwalker]

/cc @greenhat @bobbinth Let's use this as an outline for our discussion today