token-metadata: Add token-2022 support

[joncinque]

Problem

Token-2022 is undergoing final audits and almost ready for mainnet use, but token creators cannot add metadata to their token-2022 mints.

Solution

Support token-2022 in the token-metadata program. This builds on top of #931, so the first commit can be ignored. After that, here are the changes in this PR:

• support two token programs in mpl_utils. This required a local link between token-metadata and utils, so if you're happy with the changes in utils, I can add those in a separate PR, and update this PR to use some newer version once it's released.
• add *_with_token_program instruction creators that allow you to specify the token program id to use, to preserve backwards compatibility
• use token-2022 instruction creators, which supports both token and token-2022
• use StateWithExtensions everywhere to deserialize account data, which also works with both
• use test_case to test both token and token-2022 behavior in tests
• add token extensions for token-2022 tests to make sure that the full behavior is correct
• support token-2022 in the CLI
• little bits of cleanup

This PR follows the guide at https://spl.solana.com/token-2022/onchain in case you want more info. The changes are ready to be reviewed, but will require a few other pieces to fall into place before it's ready to be merged.

[cloakd]

Is this good to go or work needed? Happy to contribute if we can get this pushed through.

[joncinque]

A lot of this needs to be redone to accommodate the PNFT changes unfortunately. I can get a new PR up in a couple of weeks when I free up.

The on-chain program work is pretty simple, but I've been told that it will be a lot more work to update the tooling around it. I'm hoping that we can go piece by piece as needed, since metadata support will be extremely helpful for token-2022 adoption.

The Metaplex team has also said that they don't see a business need for token-2022 support, but I hope that will change soon!

[samuelvanderwaal]

A lot of this needs to be redone to accommodate the PNFT changes unfortunately. I can get a new PR up in a couple of weeks when I free up.

The on-chain program work is pretty simple, but I've been told that it will be a lot more work to update the tooling around it. I'm hoping that we can go piece by piece as needed, since metadata support will be extremely helpful for token-2022 adoption.

The Metaplex team has also said that they don't see a business need for token-2022 support, but I hope that will change soon!

We don't have binary space for more changes to Token Metadata right now. We're struggling to get a couple current features implemented due to this issue (e.g. Print for Programmable Print Editions). I wouldn't spend the time updating your PR until we've figured out a path forward for binary space. I do think adding support for Token 2022 for at least fungible tokens is important but it's going to be tough to get there in the near term given the current program binary size constraint.

[joncinque]

Thanks a bunch for your response @samuelvanderwaal ! I wasn't aware of the binary space issue. In that case, it might need to wait until 1.16, when solana-labs/solana#26385 hits mainnet.

In the meantime, I'm happy to explore ways to make the binary size smaller through serialization changes, removing hard-coded strings, and the like. I can also see how much token-2022 support adds to the binary size, with the sincere hope that the impact is negligible 🤞

[samuelvanderwaal]

Thanks a bunch for your response @samuelvanderwaal ! I wasn't aware of the binary space issue. In that case, it might need to wait until 1.16, when solana-labs/solana#26385 hits mainnet.

In the meantime, I'm happy to explore ways to make the binary size smaller through serialization changes, removing hard-coded strings, and the like. I can also see how much token-2022 support adds to the binary size, with the sincere hope that the impact is negligible crossed_fingers

Yeah, we've been closely following that feature, as you can imagine. Initially it was supposed to go out in v1.14 so we thought we'd have a binary solution in the near term so weren't as worried, but now that it's bumped to v1.16 we are having to look hard at Token Metadata to figure out some space savings. There's still some low-hanging fruit we're aware of such as removing static strings in e.g. logging and error messages but there are trade-offs with a lot of these types of things. The good news is we were able to remove some deprecated instructions and will be able to remove the Migration instruction once the pNFT upgrade window closes. These changes are basically just going to keep us level with finishing the final pNFT changes tho, so would be good to pick your brain some time on other ideas.

[wjthieme]

What is the status of this? Happy to help if it would help this move forward! I think this change could be a catalyst for wallets to start supporting Token22 as well

[samuelvanderwaal]

What is the status of this? Happy to help if it would help this move forward! I think this change could be a catalyst for wallets to start supporting Token22 as well

We plan to add support for Token 2022 after the Print feature is shipped (it's in final audits now), assuming we can make enough binary space for it.

[dr497]

We are very interested in the support of token 2022, thanks @joncinque for working on this!

Has any measure been made to see if there will be enough binary space?

[febo]

We are very interested in the support of token 2022, thanks @joncinque for working on this!

Has any measure been made to see if there will be enough binary space?

We started the work on this.

[febo]

Work on this can be found on the new mpl-token-metadata repository: metaplex-foundation/mpl-token-metadata#17