book 1; page 530; pre-norm formulation

[g-i-o-r-g-i-o]

Bug Report: Incorrect Pre-LN Formula

The text contains formulation 1):

1) Current (incorrect):

z = module(LN(x) + x)

If module is attn:

z = attn(LN(x) + x)

---

2) Correct version:

z = module(LN(x)) + x

If module is attn:

z = attn(LN(x)) + x

---

Why version 2 is correct:

In Pre-Layer Normalization:

1. Layer Norm is applied to the input: LN(x)
2. The result passes through the module: module(LN(x))
3. The residual connection is added after the module: + x

Version 1 places the residual inside the module's argument, which defeats both the purpose of normalization (normalizing the module's input) and the residual connection (allowing gradient flow to bypass the module).