CITATION.cff

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: >-
  Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
type: software
url: http://arxiv.org/abs/2410.17764
repository-code: https://github.com/Helmholtz-AI-Energy/frog
authors:
  - family-names: Flügel
    given-names: Katharina
  - family-names: Coquelin
    given-names: Daniel
  - family-names: Weiel
    given-names: Marie
  - family-names: Streit
    given-names: Achim
  - family-names: Götz
    given-names: Markus
preferred-citation:
  type: article
  title: >-
    Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
  abstract: >-
    The gradients used to train neural networks are typically computed using backpropagation. While an efficient way
    to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is
    biologically implausible. Forward gradients are an approach to approximate the gradients from directional
    derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused
    on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients
    and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal
    projections. We demonstrate that increasing the number of tangents improves both approximation quality and
    optimization performance across various tasks.
  keywords:
    - Computer Science - Artificial Intelligence
    - Computer Science - Machine Learning
  authors:
    - family-names: Flügel
      given-names: Katharina
    - family-names: Coquelin
      given-names: Daniel
    - family-names: Weiel
      given-names: Marie
    - family-names: Streit
      given-names: Achim
    - family-names: Götz
      given-names: Markus
  doi: 10.48550/arXiv.2410.17764
  year: 2024