Adam with B1 = 0 is RMSProp with warmup

Here is a weird thing I found while investigating the different stochastic optimizers people use for deep reinforcement learning: the Adam optimizer with the B1 hyperparameter set to 0 is the same as the RMSProp optimizer, with an exponential learning-rate warmup schedule.

To show this, I will write out the Adam update rule in full, and with B1 set to 0. Note that I will use a slightly different convention than in the original Adam paper that has the debiasing terms factored out and multiplied later to show this, as used here but the actual update rule is the same [credit to the authors of that playbook for the written-out algorithms as well].

Above - full Adam update rule.

Above: Adam w/ B_1 = 0.

Now, examine the full RMSProp update rule:

It is trivial to see from this that if you set gamma=0 in the RMSProp term (which is equivalent to RMSProp without momentum, or how it is used by default) than the update rules are identical, other than the multiplied root(1-B2^t+1) term (obviously B2 and p are named differently, but are functionally the same term). But given that B2 < 1 (it is usually set to ~0.999) this term will asymptotically approach 1 as t-> infinity.

Above - plot of the debiasing term for B2=0.5. You can see that while it starts small at 0, it approaches 1 as the number of iterations increases.

The only other small difference is in initialization - RMSProp initializes the v term to 1, while Adam initializes the equivalent v term to 0. But this is relatively small and can differ by implementation anyways - for example, PyTorch’s Optim library initializes v as 0 for both Adam and RMSProp.

My argument is that this debiasing term we see in Adam can be seen as a form of learning rate warmup (when B1=0), which is a thing sometimes done in deep learning where the learning rate is set very low and increases over the first couple of iterations before another scheduler like cosine annealing is applied.

For the default value of B2 = 0.999, the bt = 0.99 at t ≈ 4000, and for B2 = 0.99, bt = 0.99 at t ≈ 400. These are both in the range of how many update steps are typically performed using learning rate warmup.

One might ask why you should care about this considering virtually no one uses Adam with B1=0, but there is at least one pretty high profile case of people doing this: the DeepMind London teams’ RL Starcraft II agent [1] was trained using Adam with B1=0 and B2 = 0.99, both for its value head and policy gradient update (using PPO for the estimated PG). So it is possible this learning rate warmup might very partially responsible for some of its success. I think it is probably more likely they simply did not try RMSProp/that it would’ve performed equivalently.

As far as I can tell this is a novel observation. I found two other people on the internet that have noticed something similar. Amirhossein Rezaei said B1 = 0 exactly replicates the RMSProp step which is slightly incorrect because as I explained above the debiasing term does affect the first several update steps [2] and Nishant Nikhil argued it was the same as RMSProp with bias correction [3] which is technically correct and another way of examining this phenomenon. Essentially, the new thing to note is that when you are not correcting for the bias of both the moving average and squared average, the bias correction on the squared average becomes like learning rate warmup.

Keywords: Stochastic Optimization, Deep Learning, Machine Learning, Gradient Descent, ADAM, RMSProp.

References:

[1] Vinyals, Oriol, et al. “Grandmaster level in StarCraft II using multi-agent reinforcement learning.” Nature 575.7782 (2019): 350-354.

[2] https://datascience.stackexchange.com/questions/117018/does-settings-beta-1-0-or-beta-2-0-means-that-adam-behaves-as-rmsprop

[3] https://medium.com/@nishantnikhil/adam-optimizer-notes-ddac4fd7218