efficient implementation

[amrhamedp]

Hi,
I wonder if relative distances and coordinates can be handled more efficiently using memory efficient attention as in " Self-attention Does Not Need O(n^2) Memory". It is straightforward for the scalar part.

[lucidrains]

yea definitely

have you found En-transformer useful?

[amrhamedp]

yes ...it is pretty useful however I believe that distances should be used to allow nodes positional encoding. I mean something like IPA (invariant point attention) or schnet like encoding for distances using RBF.