Does the current IA3 include the two additional loss terms?

[Vincent-Li-9701]

I was going through the source code of IA3, but it doesn't seem to contain the two additional loss terms proposed by the author. unlikelihood loss (Lul) and a length-normalized loss (Lln). Am I missing anything here? Could someone point to me where it's implemented?

[SumanthRH]

Hi,

IA3 - “Infused Adapter by Inhibiting and Amplifying Inner Activation" is simply the PEFT method that adds additional parameters. You seem to be referring to the loss functions in the full T-Few recipe. You should have a look at the official implementation from the authors here

[Vincent-Li-9701]

I see. Thanks. I was checking the source implementation it seems that the ia3 in peft doesn't include the recipe. I imagine we would have to implement that on our own.

[SumanthRH]

Yes that's right! Custom loss functions don't seem to fit in with the goal of the PEFT library (which is why it only implements IA3, not the full training recipe in T-few), so I don't think this is coming anytime soon. I will still cc @younesbelkada for a final word.

[younesbelkada]

I second what @SumanthRH said, building and maintaining custom loss functions is currently not really in the scope of PEFT, but I think if there is a huge community interest we might consider that in the near future! cc @pacman100 @BenjaminBossan

[BenjaminBossan]

Would that perhaps be something that could be added to transformers Trainer? I would like to avoid adding training code to PEFT proper (it's okay for examples).

[SumanthRH]

Interesting! Does Trainer actually allow for different loss functions? From the source code for compute_loss , it looks like the Trainer simply uses the default cross entropy loss, and any customization requires subclassing compute_loss. Maybe this should be kept to an example notebook in examples then @BenjaminBossan ?

[BenjaminBossan]

If it's possible without too much extra code to add an example with the criterion used by T-few (haven't closely checked it), that would be a nice addition. It also would be nice to have a comparison between that and the default loss.

any customization requires subclassing compute_loss

That would be a possibility. From my understanding, it would also be possible to modify the module's forward method to return a dict, one item of which should be "loss". This could be more convenient if a custom forward needs to be implemented anyway, otherwise subclassing Trainer seems to be more straightforward.