Training methods for rpuconfig

[kusionlion]

Hello!

I am trying to train the BERT model using different weight noise modifiers.

In the example 24, ADD_NORMAL was used with a std_dev specified in the wandb. I tried to swap the modifier type to ADD_POLY and give a list of coeffs. But it doesn't seem to work because no matter what coefficients were used, the training losses are pretty much the same. Please help!! Thank you!!

These are what I changed.

1. set the ARGS.wandb=True

2. changed the modifier_noise to a list of coefficients
   if ARGS.wandb:
   import wandb

   SWEEP_CONFIG = {
   "method": "random",
   "name": "modifier noise sweep",
   "metric": {"goal": "maximize", "name": "exact_match"},
   "parameters": {"modifier_noise": {"values": [[1,2,3],[2,3,4],[3,4,5]]}},
   }

3. changed the rpuconfig type and coeffs.
   rpu_config = InferenceRPUConfig(
   clip=WeightClipParameter(type=WeightClipType.FIXED_VALUE, fixed_value=1.0),
   modifier=WeightModifierParameter(
   rel_to_actual_wmax=True, type=WeightModifierType.ADD_POLY, coeffs=modifier_noise
   ),

[maljoras]

I believe you have to set std_dev as well when using the coefficients as it acts as an additional scale.

[kusionlion]

Thank you!! It works now.

[kusionlion]

What does the std_dev scale?

If I am understanding correctly, the polynomial function specifies the width of a normal distribution that is added to various weight values during HWA training. Where does std_dev comes in effect?

Also, what does this expression mean?
" \sigma_\text{wnoise} "