-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rotation equivariant MLP for 2d images #19
Comments
@amoskalev I don't think SO(2) has a well-defined action on images as pixel permutations. If you rotate a picture, some pixels will be lost (out of boundary) and it is no longer a group action. SO(2) does act on R^2 but this is a different story (which seems to be what the author's paper was addressing). Also a question for the author @mfinzi, is there a reason why you only consider equivariant linear forms? Noether's finite generation of ring of invariants is telling us that if we want to get closer (in fact exact if we only consider algebraic functions) to universal approximators of equivariant functions, it is possible to also consider/compute higher order invariant functions if the group is not too wild (works for all groups considered in your paper). Computations for the full invariant polynomial basis might be impractical (exponential complexity), but maybe including some higher order ones helps? |
Ah sorry guys, I missed this github issue. To avoid the boundary issue and have a well defined action, I think you would need to define the representation through bilinear interpolation with e.g. periodic boundary conditions (this way the pixels are not lost). As in \rho(g)v where v is the flattened image and the bilinearly interpolated rotation operation defines \rho(g). Bilinear interpolation does act linearly on the image, so you would just need to verify that with the appropriate boundary conditions the \rho(g) rotation operation is indeed invertible and composes as you would expect \rho(g)\rho(h) = \rho(gh) which I believe would be the case if constructed appropriately but might require some care. You would want to define the bilinear interpolation \rho implicitly using the |
yeah that's a solution for a lot of group actions on image grids. But maybe not for SO(2). If you can act SO(2) by permutation that means there is a map between SO(2) and S_{n*m}, but the latter is discrete and the former is a connected Lie group so the map has to be trivial. |
And @honglu2875, in the framing of the paper the focus is on the linear equivariant maps, but in https://emlp.readthedocs.io/en/latest/notebooks/6multilinear_maps.html there are some examples with finding the space of multilinear equivariant maps, though as you mention doing so can quickly become computationally expensive for higher order functions. In my understanding though, the solution basis for equivariant multilinear maps e.g. the bilinear map V_1 \times V_2 \rightarrow V_3 (or written differently V_1 \rightarrow V_2 \rightarrow V_3) can be mapped over from the solution basis for the equivariant linear map (V_1 \otimes V_2) \rightarrow V_3. In the EMLP network, we use a Bilinear layer so as to make it possible to form these higher order equivariant functions. If you could find an explicit example of an equivariant polynomial function that could not be expressed exactly in EMLP or a more general equivariant function that could not be approximated by EMLP, I would be interested. |
And @honglu2875 , this wouldn't be a pure permutation representation so I'm not sure that logic applies. Rather because of bilinear interpolation, most rotations would not be permutations, only a select few like rotation by 90 degrees. Instead it's its own kind of unusual action on the image |
Ooooh I read too quickly. I saw periodic boundary conditions and thinking you just wrap pixels around. Yeah I agree some sort of interpolation seems to be able to introduce nontrivial representation. |
Oh this is very nice! Just to make sure I don't misunderstand, can I assume Bilinear layer is to go through invariant degree 2 polynomials? |
Hello Marc, thanks for your work!
Using your library, I want to implement a rotation equivariant MLP and apply it for roto-MNIST classification, where I rotate 2D digits and ravel them into a vector before feeding them into the network.
Can you give an advice of how to implement EMLP model for such a problem?
Would it be:
model = nn.EMLP(repin,repout,G)
with repin = 2828Scalar ; repout = 10*Scalar ; G=SO(2)?
Thanks.
The text was updated successfully, but these errors were encountered: