Tips for applying Muse Attention

[zhaoguangxiang]

Hi, I'm the author of the paper Muse Attention. Thank you for your reproduction. Since Muse has several modules including FFN CNN Self-attention and Cross Attention in one residual block, the network becomes more shallow (regarding the residual blocks) and wide. Reducing the dimension of each unit by 1/3, using a smaller initialization (e.g., from Xavier init to torch init), and increasing the number of residual blocks (while keeping the number of parameters) will help improve performance.