create configuration file for MaMMUT training (#521)

Summary:
Pull Request resolved: #521

Mostly based on original coca and https://github.com/lucidrains/MaMMUT-pytorch

Update the logics of loading checkpoint for MaMMUT text decoder as well.

Differential Revision:
D52891614

Privacy Context Container: 303860477774201

fbshipit-source-id: 192a1826fd59a80bf99e8545408e19938069a599

torchmultimodal/modules/layers/transformer.py

@@ -412,10 +412,7 @@ def _forward_prenorm(
  self_attn_output = attn_output + hidden_states
 
  # Optional cross-attention
- if self.use_cross_attention:
- assert (
- encoder_hidden_states is not None
- ), "encoder_hidden_states must be provided for cross attention"
+ if self.use_cross_attention and encoder_hidden_states is not None:
  assert hasattr(
  self, "cross_attention_layernorm"
  ), "Cross-attention layernorm not initialized"