Skip to content

Commit

Permalink
Fix MNIST training example with dynamic L2 rescale logits. (#76)
Browse files Browse the repository at this point in the history
Training set accuracy 0.96670
Test set accuracy 0.93730

Note: test set accuracy can change slightly if using an `eps` in dynamic rescaling.
  • Loading branch information
balancap authored Jan 9, 2024
1 parent 23a74b1 commit 621d85e
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions experiments/mnist/mnist_classifier_from_scratch.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ def predict(params, inputs):

final_w, final_b = params[-1]
logits = jnp.dot(activations, final_w) + final_b
# Dynamic rescaling of the gradient, as logits gradient not properly scaled.
logits = jsa.ops.dynamic_rescale_l2_grad(logits)
return logits - logsumexp(logits, axis=1, keepdims=True)


Expand Down

0 comments on commit 621d85e

Please sign in to comment.