PoliBa-Courses/Deep-Learning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Deep-Learning/Chapters/5-Optimization/Fancy-Methods/LION.md
Christian Risi 26e0f11b42 Added Lion
2025-04-20 11:59:16 +02:00

1.7 KiB
Raw Blame History

Lion (evoLved sIgn mOmeNtum)1

Lion is a genetic search algorithm aimed to find the best optimizer.

It starts from a population of AdamW algorithms to speed up the search. Opposed to Adam and AdamW, it keeps track only for the momentum and gradient sign, requiring less memory.

Since uniform updates yields larger norms, Lion requires a smaller learning-rate and a larger decoupled weight-decay $\lambda$2.

The advantages of Lion over Adam and AdamW increase with the size of the mini-batch2

Symbolic Representation3

New trained algorithms are represented simbolically, bringing these advantages:

  • Algorithms must be implemented as programs
  • It easier to analyze, comprehend and transfer to new task these algorithms, rather than other algorithms such as NeuralNetworks
  • We can estimate the complexity by looking at the length of code

Tournament4

The best code is found with a tournament style evolution. Each cycle it picks the best algorithm which will be copied and mutated and the oldest is removed