Skip connection in encoder and decoder

[Xiaohan-Chen]

Hi, thanks for sharing this wonderful project. I notice that in the transformer encoder, there is a skip connection between the encoder output and input, i.e. src = self.encoder(src) + src_start.
It seem different from the original transformer model? Decoder also has the same connection.

def encode_src(self, src):
    src_start = self.input_projection(src).permute(1, 0, 2)

    in_sequence_len, batch_size = src_start.size(0), src_start.size(1)
    pos_encoder = (
        torch.arange(0, in_sequence_len, device=src.device)
        .unsqueeze(0)
        .repeat(batch_size, 1)
    )

    pos_encoder = self.input_pos_embedding(pos_encoder).permute(1, 0, 2)

    src = src_start + pos_encoder

    src = self.encoder(src) + src_start

    return src