-
Notifications
You must be signed in to change notification settings - Fork 19.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ops.random.alpha_dropout
and layers.AlphaDropout
#18940
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #18940 +/- ##
==========================================
+ Coverage 79.51% 79.57% +0.05%
==========================================
Files 336 337 +1
Lines 34975 35056 +81
Branches 6865 6872 +7
==========================================
+ Hits 27812 27896 +84
+ Misses 5585 5583 -2
+ Partials 1578 1577 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
keras/backend/jax/random.py
Outdated
@@ -81,6 +82,24 @@ def dropout(inputs, rate, noise_shape=None, seed=None): | |||
) | |||
|
|||
|
|||
def alpha_dropout(inputs, rate, noise_shape=None, seed=None): | |||
noise_shape = _get_concrete_noise_shape(inputs, noise_shape) | |||
alpha = 1.6732632423543772848170429916717 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any specific reason for having more than 6-7 decimal points?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In practice it will be cast to float32 so some precision will be lost. The number above is just taken from the paper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! Ok to add the layer -- but let's drop the op.
|
||
def call(self, inputs, training=False): | ||
if training and self.rate > 0: | ||
return backend.random.alpha_dropout( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can implement the layer in terms of backend ops and random.uniform
. There is no real need for the alpha_dropout
op -- only the layer would get used (even then, it's fairly niche usage).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I've made the changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the contribution!
Related to #18467
This PR should benefit the users who want to apply the self-normalizing property to their models.
The implementation is taken from:
https://github.com/keras-team/keras/blob/v2.14.0/keras/layers/regularization/alpha_dropout.py#L28-L104