Skip to content

Commit

Permalink
Update the short ReLU description
Browse files Browse the repository at this point in the history
don't call it one-to-one operation
  • Loading branch information
serhii-karelov authored and xiaohk committed Oct 14, 2023
1 parent 119e394 commit d0971f9
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/article/Article.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@
Neural networks are extremely prevalent in modern technology&mdash;because they are so accurate! The highest performing CNNs today consist of an absurd amount of layers, which are able to learn more and more features. Part of the reason these groundbreaking CNNs are able to achieve such <a href="https://arxiv.org/pdf/1512.03385.pdf" title="ResNet">tremendous accuracies</a> is because of their non-linearity. ReLU applies much-needed non-linearity into the model. Non-linearity is necessary to produce non-linear decision boundaries, so that the output cannot be written as a linear combination of the inputs. If a non-linear activation function was not present, deep CNN architectures would devolve into a single, equivalent convolutional layer, which would not perform nearly as well. The ReLU activation function is specifically used as a non-linear activation function, as opposed to other non-linear functions such as <em>Sigmoid</em> because it has been <a href="https://arxiv.org/pdf/1906.01975.pdf" title="See page 29">empirically observed</a> that CNNs using ReLU are faster to train than their counterparts.
</p>
<p>
The ReLU activation function is a one-to-one mathematical operation: {reluEquation}
The ReLU activation function is an elementwise mathematical operation: {reluEquation}
</p>
<div class="figure">
<img src="PUBLIC_URL/assets/figures/relu_graph.png" alt="relu graph" width="30%" height="30%"/>
Expand Down Expand Up @@ -286,4 +286,4 @@
Georgia Tech and Oregon State. We thank Anmol Chhabria, Kaan Sancak, Kantwon Rogers, and the Georgia Tech Visualization Lab for their support and constructive feedback. This work was supported in part by NSF grants IIS-1563816, CNS-1704701, NASA NSTRF, DARPA GARD, gifts from Intel, NVIDIA, Google, Amazon.
</p>
</div>
</body>
</body>

0 comments on commit d0971f9

Please sign in to comment.