Order of Norm vs. function f

[madaan]

Thanks, the playbook looks pretty cool!

I am curious about:

Normalization should be the last operation before the residual. E.g. x + Norm(f(x)).

Is this advice for specific settings/norms?

For modern LMs, the order typically is x + f(Norm(x)). For example, transformer blocks in language models typically have the following design:

def block(x):
    # x is the input, ln{1, 2} are layer norms, attn is self-attention, mlp is a feed-forward network
    return x + mlp(ln2(x + attn(ln1(x))))

Some examples are T5, GPT-2, and I think PaLM also applies LayerNorm before MLP/Attention.

[jmgilmer]

Thank you for pointing this out. This is a typo on our end!

[madaan]

Ah ok, thanks!