Structure for backwarding gramians

[PierreQuinton]

It appears to be important to be able to bakward Gramians of Jacobians through some Sequential. It is thus important that we think about how to do that.

I believe we could in principle have a backward_gramian method that takes as input the vector to backward (the output of the module) and the Sequential. It would then go through each layer, compute the Jacobian and accumulate the Gramian.
From this we could extract weights (with a weight extractor that maps a Gramian to weights) and backpropagates them using a normal call to autograd.backward.

This may not be the best way to proceed, so we need further discussions.

[PierreQuinton]

Here is a recap of the things I have discovered while exploring:

• torch assumes that the graph is a tree, so two nodes that accumulate in a are considered distinct, this is because of reflexivity of addition. For accumulating a Jacobian, it is also a tree, but for computing the Gramian, then it is not (Gramian of sum of Jacobians is not sum of Gramian and there is no way to make it much simpler I think). So playing with the graph itself would imply things as counting arrows to leafs, computing Gramian only when the Jacobian itself is ready, there is a possibility of doing topological ordering of operation so as to compute Gramian the earliest. This is a whole new AD engine and we cannot really do that yet, in particular when the structure is quite flat, such as a Sequential, then those operations have a nice form such as the one in our paper (v2 or more).
• If we assume sequential structure, then we could still rely on playing with the graph but there is no point anymore, we could implement the sequential ourselves, store all the accumulation and do the backward phase ourselves, this would imply that our model is responsible for the forward phase (compute and store activations/output, return output), reverse accumulate the Gramian (Algo 3 of the paper), and then why not, the backward phase. I think it would be nice to have a class for representing modules on which we can accumulate Gramians, some would do it reverse, we could then have a wrapper of a Module and a Sequential that maps a sequence of normal module to one that is the sequential that also can reverse accumulate.