Injecting PCM noise

[maicoldubbio]

Hi!

I'm looking into WeightModifierType.PCM_NOISE and I have a few questions:

1. After starting some trials, it looks like it's only available on GPU. Any chance to get it working on CPU as well?
2. Which noise is exactly carried in? Is it both read and prog noise? How is drift variation considered? By setting e.g. read_scale, will noise intensity be increased? If you could give some formulas it's be great!

Thanks a lot.

[maljoras]

Hi @edminge,
Indeed the PCM_NOISE option is only available on GPU. However, the PROG_NOISE option and POLY are both available. For those any programming noise polynomial can be used, and this only lacks the ability to define fraction of corrupt devices (at set and random states) and the drift. We found that drift is not really useful for HWA training so this is enough for most cases. Note that prob at zero can be set in all cases with the pdrop parameter.

[maicoldubbio]

Thank you @maljoras .

Let me ask: can I set the very same WeightModifierParameter.coeffs I have for my program_noise polynomial (thus as a function of g_t)? Because I see they're applied in terms of w_ij, but assuming rel_to_actual_wmax=True, it should be 1 to 1.

[maljoras]

The exact definition of the noise during hardware aware training can be found in the paper Nat. Comm 2023 Rasch et al.

[maicoldubbio]

Bonus question: what is the effect of remapping weights during HWA training? I see in Nat. Comm 2023 Rasch et al./Suppl. Material this is done often every N mini-batches, however why is this useful?

I guess the tile.remap_weights() is being referred.

[maljoras]

The remapping is ensuring that the maximal conductances that are used to represent the weight (which we model as normalized values between e.g. -1.. 1) are as large as possible e.g. close to gmax (the maximal conductance one wants to use on the analog device based on physical considerations). Note that the "logical" weight is a combination of analog parts (conductances) and the output scaling, which we both train separately. The remapping ensures that the analog parts of the weight is as large as possible, and this as the larges signal to noise ratio.

[maicoldubbio]

Thanks for the reply.

Can you elaborate on that? I definitely see the benefits, but why isn't that done automatically? Moreover, in the previous version (0.8) how was that handled and did it have less performance? Since I see this has been added only in 0.9.

I wonder if that could be done similarly in the previous version.