#

entropy-collapse

Here is 1 public repository matching this topic...

Nandan91 / relu-revival-normfree

PyTorch implementation of normalization-free LLMs investigating entropic behavior to find desirable activation functions

pythia leaky-relu relu privacy-preserving-machine-learning pytorch-implementation gelu gpt-2 model-optimization transformers-models normalization-free-training llm-inference llm-evaluation llm-architecture private-inference entropy-collapse attention-we

Updated Nov 2, 2024
Python

Improve this page

Add a description, image, and links to the entropy-collapse topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the entropy-collapse topic, visit your repo's landing page and select "manage topics."