Noisy Net Issue

[neal2164]

DI-engine/ding/torch_utils/network/nn_module.py

Line 713 in 580ea65

return F.linear(x, self.weight_mu, self.bias_mu)

This looks like a bug. The purpose of noisy net is to improve exploration. Disabling noise injection for the collect_model defeats the purpose. From my understanding, noise injection should be only disabled for the eval_model.

Also see "Concerning the action choice, we generate another independent sample ε′′ for the online network and we act greedily with respect to the corresponding output action-value function" from https://arxiv.org/pdf/1706.10295 for reference.

[PaParaZz1]

The issue you raised appears to be a usage bug, where noise injection should be only disabled for the evaluation model. We will fix this problem in the next week and run some validation experiment on Atari.

[neal2164]

I believe you are right.

BTW, thanks for your team to work on and share the nice platform:)