doubt about patch embeddings

[zaocan666]

Hi~awsome repo.
But I wonder if it is necessary to implement GELU and LayerNorm after linear layer to get patch embedding. Neither the ViT paper and code applies these layers.
What I mentioned is in Line 269 and 270 in /SETR/transformer_model.py

[920232796]

transformer 文章里面是使用了这两个操作，你可以去看下原文中的模型结构。 Attention is all you need.

[zaocan666]

但ViT的文章和代码好像没有用

[920232796]

这个我也不清楚了 说实话我没看过那篇文章跟代码... 你可以去掉试试 我觉得影响不大应该。

[zaocan666]

好的，多谢