The background reasoning on why Molecule was created

[doitian]

Originally posted in Telegram

Jordan Mack, [May 20, 2020 at 8:46:32 AM]:
Hey Xuejie, can you tell me the background reasoning on why Molecule was created? @xxuejie

[xxuejie]

This is a very long topic. Basically I see 2 major points in molecule:

1. Partial loading: given a serialized block from the network, will we be able to quickly locate, for example, the lock script of the 1st output belonging to the 3rd transaction? This might not look like a big deal, but it in fact has huge potentials:
   1. It means we don't have to parse every single message we received from the network, and allocate more tiny structs, vectors or other data structures to keep them. We can instead just represent them as a slice of bytes read off the NIC, then each individual component in the block, can then just be a slice into the bytes, in our current implementation, those slices all share the fundamental bytes, memory allocations have been regulated and minimized this way, we are dealing with much less pressure when it comes to malloc, which can be quite a hassle for a modern multi-threaded network application.
   2. This also significantly improves the design of smart contracts on CKB: with a format that supports partial loading, the smart contract can just load the specific few bytes it needs. For example, even if we have a syscall for loading the whole transaction, this doesn't mean we have to load the full transaction every time. A properly designed molecule library can just first load a few bytes of the transaction header, then depending on what specific component the smart contract requires, loads those required bytes only to save both memory in VM, as well as cycles consumed.
2. The second issue here is that hashing consistency: when we are creating transactions with the same data in different places, most likely also attaching components in different order, are we sure the serialized bytes are all the same, so the hash of the bytes also stays the same?

If we look at the existing serialization formats, some satisfy the first point(e.g, capnproto, flatbuffer), some satisfy the second point(RLP, Bitcoin's custom format), but we failed to see one that unites the best of both worlds. That's why we started the task to design molecule here.

[doitian]

We originally decided to define a strict version on flatbuffers. We called it cfb. We finally switched from cfb to molecule because flatbuffers is too complex to reason about and has too many corner cases to make the format canonical. Another motivate is that we want the serialization format simple enough, so the parsing and generating is also very efficient in CKB VM.