Question about constrained problem solving.

[DoTulip]

Hello, everyone!
I am troubled by a problem and want to ask for your help. I'm doing some research with neural networks, and the objects I'm studying have some constraints (both equality and inequality constraints). Optimizers like Adam only work for unconstrained problems, what can I do to elegantly handle these constraints (I tried the penalty function method, but it doesn't seem to work very well)?

[ramithuh]

Hi @DoTulip,
I have worked on some optimization problems where the parameters have admissible regions (eg: 0 < param < $2\pi$) that they can lie on, is your problem something like that?

[DoTulip]

Hi @ramithuh,
Thank you for your reply!
Yes, I encountered such a constraint. In addition, there are other constraints that are imposed on the objective function.

[ramithuh]

I see..
I'm not sure whether this will exactly be helpful in your case, but for me I've been dealing with the sigmoid activation a lot when I need to constraint a parameter to a particular range (I've used this a lot in physics based optical neural networks), and it worked fine. (I had to have a good schedule for the temperature however)

Based on this idea of controlling the temperature, there have been many efforts in implementing quantization as well [1,2]. Idea is to start with a soft function and gradually make it rigid.

[1] - https://arxiv.org/abs/1908.05033
[2] - https://arxiv.org/abs/1911.09464

weight = nn.sigmoid( temperature * shadow_weight) * const

(the video below is an example of quantization through summing up shifted sigmoid functions )

Screen.Recording.2022-09-12.at.09.12.16.mov