RL isn't dead because of the Decision Transformer

This is a direct response to this blog post so you should probably read that first before reading this. It’s a Medium article and so is paywalled, but you should be able to use this Google cache to read it.

The main argument Austin Starks makes here I want to critique is that RL is made obsolete by the Decision Transformer model, a different model/way of approaching Markovian decision making problems. I will extol the virtues of RL less, other than a brief mention of some high-profile successes. I will mostly focus on his main arguments, and discuss more minor issues later.

This will be separated into 4 sections - an overview of the Decision Transformer, my core argument against Starks’, responding to some of the stuff in the article proper, a short conclusion, and then an appendix where I nitpick details.

1: Decision Transformer Overview

First, I wanted to give a brief overview of what the Decision Transformer does, because the top level idea of “treat Markovian decision making as a sequence modeling problem” isn’t self-revealing.

The important things distinguishing it from normal RL is that A] in its vanilla form, its an offline learning algorithm that requires full rollouts to learn B] it tries to self-supervized learn the sequence of states, actions, and return in a MDP like how LLMs learn language as a sequence of words/tokens.

My first reaction to that was “doesn’t that just turn into a fancy form of learning a model of the MDP which doesn’t learn to solve it”. The trick is that they don’t model reward but reward to go/the return (I will call this the “return” or RTG as short for “reward to go”), which brings that key element of the temporal decision making procedure into the process. They don’t model the return of the agent’s current policy (like the value head in actor-critic methods) or the optimal policy (like in value iteration) but just whatever the trajectory that the sample was a component of’s return. Toy example - I have two states, I start in S1, take A1=leave state with R = -100, then I stay in S2 forever regardless of action (let’s say I only have one, A1) with R = 10, with discount gamma = 0.9. My rollout looks like this: [S1, A1, R=-100, S2, A1, R=10, S2, A2, R=10, …] .

My first RTG is R1 + gamma^1 * gamma^2 * R2 + gamma^3 * R3 + … + gamma^inf * R_inf, which becomes = -100 + 0.9 * 10 + 0.89 * 10 + …. = -100 + (10/(1-0.9) = 0. Then my second rollout and onwards is just the geometric sum component so becomes 100 forever for however long my rollout goes. I found the theoretical RTG closed form here by taking the geometric sum limit, but in practice the RTG would be estimated empirically, so I would start at my current state, add the current reward, add the next reward discounted by gamma, next by gamma^2, etc. until I’m done, and then repeat for each state. For reasons that will make sense in a second, when they turn rollouts into a sequence to learn auto-regressively, they put the return first, then the state, then the action. So my final sequence to plug into my Decision Transformer would look like [0, S1, A1, 100, S2, A1, 100, S2, A1, …]. During training, the Transformer learns to predict the next RTG, state, or action given the history it is presented, which it learns probabilistically from its training data.

At inference time, you use your auto-regressive model but condition on high return. So I have my history RTG1, S1, A1, RTG2, S2, A2, etc., and am now currently at SN trying to make a decision on what action AN I should take. I condition my reward RN high (let’s say 100 ) and add it to my sequence, then take the most likely action from the output of my Transformer. So the sequence I would give the transformer looks like [RTG1, S1, A1, … [all the other history], … RTGN-1, SN-1, A-1, 100, SN] and the Transformer would give me back a distribution over actions, which I take the most likely of or do some probabilistic sampling thing to get AN. Very cool! There are some other technical details I’m leaving out but this is the main gist.

My initial impression was “isn’t this just a fancy form of Imitation Learning (IL)?” The authors clearly thought so too because they included a whole section about that in the paper (section 5.1) where they argue it is doing something better because you can effectively feed a Decision Transformer a bad rollout, but conditioning on high RTG will stop it from giving you a bad policy, wheras BC will just accept that the bad rollout is how it should act and tell you to act like that regardless. This is a nice result - showing that by doing a form of environment modeling instead, you can have both good and bad data but still output a good policy.

2: The Decision Transformer doesn’t solve RL

My core argument against the Decision Transformer (I’ll call it the DT from now on) quote unquote “solving RL” is the following: you need access to high-quality rollouts for it to work, which you can (sometimes) only get from true online RL. This is true both theoretically and practically - if you think about how the DT learns, at some point in its training data it needs information about high RTG rollouts, otherwise when you condition on them at inference time its predictions probably won’t be all that strong because you’ll be asking it about a part of the sequence space it is unfamiliar with. One of the cool things about online RL is that a good RL agent has some notion of exploration, so it should be able to find good trajectories all by itself. This is not true of the DT, which you need to feed at least some high-quality data. They do this in the paper (for the complex Atari and OpenAI Gym tasks) by taking data from well-trained online RL agents for the rollouts, both at its peak performance and from its less-optimal replay buffer from training. This should immediately make the argument that the DT is all you need strange - even in their own paper they have to use online RL algorithms to find the data needed to train the DT! Some of this I suppose comes down to philosophical differences between online and offline RL, and there are modifications to the DT to use it online, but the core point remains that the really difficult task in RL of learning to explore and then exploit an environment is not really something the base DT does.

My argument also seems consistent empirically based on the results given in the DT paper. One of the ablation studies they did was to compare the DT against high% Behavior Cloning, where their BC network was just given a smaller amount of high-quality data to emulate. In high-data regimes like the OpenAI Gym or the Key-To-Door environment they found 10% BC outperformed the DT (Table 3).

They also found that the highest RTG the DT could give a trajectory for was quite correlated with the highest RTG trajectory in the dataset (Figure 4) which confirms the idea from before - you cannot just throw lots of environment data at a DT and ignore quality, you have to have at least some high-RTG trajectories to generalize well.

One might ask why I would care if I need high quality trajectories - every other area of ML is very concerned with high-quality data, maybe I should just accept I need to find some. The trouble is that this becomes difficult in areas where just sampling randomly is ineffective, or where humans can’t provide high enough quality trajectories - for instance, if I wanted to make a superhuman Dota II agent, it would probably be insufficient to just use lots of human trajectories because more likely than not the DT would be limited by the maximum RTG of the human players in the dataset. So we still need online RL to help with exploration.

It is also worth noting even on the Arcade Learning Environment, it doesn’t even beat a temporal-difference based offline RL algorithm (CQL). So its empirical superiority is also not a shut case.

One more problem DTs have is that they do not handle environment stochasticity well. I won’t go into too much detail, but will give a simple example. Imagine we are in State 1, and have two actions - action 1 has a .0005% chance to bring us back to state 1 with 10 reward, and a 99.9995% change to bring us back to state 1 with -900000 reward. Action 2 has a 100% chance to bring us back to state 1 with 5 reward. If we used the DT and simply conditioned on the highest reward we know is possible (10) the DT would pick action 1 all the time because of the trajectories it sees, the only ones which follow a upcoming reward of 10 are the ones that pick action 1. But the average reward of action 1 is catastrophically bad (just above - 900000) compared to action 2 (5), and any rational agent should pick action 2. There is work that addresses this, but given how common stochastic environments are in the real world this is a pretty bad deficit to have.

3: Responses

Again, I will try to just respond to the core criticisms here. I will address other things in the fourth section.

1] “Deep RL doesn’t work for stock market prediction”

There are cases where RL does not work due to no fault of its own, and one of them is in data-limited environments or cases where the environment model is poor. One of the assumptions made in RL (a markovian decision making process) might not apply to a stock market because there are other agents who respond to your computer agents’ policy, making the transition distributions non-stationary.

It is also worth noting there are many cases where RL might work, but it is just not quite the strongest tool. For the stock market case, people have been thinking about stocks for quite a long time and have developed sophisticated market models to forecast prices and such, and it is hard to compete against that strong prior knowledge with an algorithm that has to learn it all from scratch. If you think about successful stock strategies as a gigantic evolutionary algorithm that has been running for at this point decades, funded by one of the most lucrative businesses on the planet able to attract the top mathematical and computing talent, it is not surprising they have discovered dynamics in the stock market that are useful in developing good strategies. Again, RL agents just have one core assumption: they are working in an MDP. They don’t get prior knowledge about buying low and selling high, or how Indonesian weather patterns affect palm oil futures, or any of that.

So in my view this doesn’t make RL suck, it just makes it worse than other things sometimes. But that is ok - when you use a very general solution, sometimes you will give something up compared to domain-specific bespoke solutions. These general solutions shine most when the domain-specific ones aren’t very good (like in playing complex video games) or are unknown (like in RLHF).

2] Grab-bag complains

Stark goes through a couple of common complains in one paragraph. I’ll quickly address them here:

a] Computational Expense/Sample Inefficiency: This is obviously true but there have been strong steps recently to reduce these. Model methods like DreamerV2, applying the right tricks to value methods like CrossQ or BBB, and the right modifications to policy methods like SAC can get you agents that are starting to rival human sample complexity with strong wall-clock performance. Some of these are sample efficient enough to use in the real world and can train online - I went to an RL symposium recently where Dr. Antonin Raffin discussed how RL was now sample efficent enough to enable limited training in the real world for robotics. Another good example is training an RL agent to be competitive in Gran Turismo, entirely on the real game running on real Playstations in realtime.

But there’s also a more principled critique of the mindset that computational expense or sample inefficiency are even problems. Dr. Rich Sutton’s famous essay The Bitter Lesson compellingly argues that the best methods in AI are always ones that can scale well with compute, given that compute is the one inherent advantage computers have over humans or other animal agents. It is probably OK that it took lots of GPUs and months of training for PPO to beat people at Dota - because computers are powerful, and modern RL methods are both scalable (somewhat) and parrallelizable (somewhat) they could beat humans on a task it takes years to learn in just a couple months! Finding methods to leverage advantages where computers are better than people should be our goal.

b] Convergence issues - this one is somewhat harder to seriously argue against, but I will say that there is also a lot of work into this area. Maybe the most popular RL algorithm today, Proximal Policy Optimization, was developed as a way to try to guarantee monotonic policy improvement in the policy gradient setting. It is also worth noting that the authors proposed solution (the DT) doesn’t really avoid this - there are tasks in the DT paper that it fails to solve optimally, and generally training large Transformer models, while easier than RL models, has its own problems. It’s well-known the training of GPT-4 had to be paused and restarted a couple times because it diverged or encountered other training problems, for instance.

3] The DT solves everything

Obviously Section 2 responds most directly to this, but one other note I’ll make is that it’s not super clear to me that Transformers are any more or less intuitive than Deep RL. I am much more confident in my ability to stand in front of a whiteboard and explain how every single detail of Q-Learning works, down to the grittiest mathematical detail, than take even a wide stab at how the Attention mechanism works.

4: Conclusion

The Transformer model/the attention mechanism are obviously quite powerful, it would be hard to deny that. But the idea that it will replace everything else in ML seems misguided to me, maybe by hype, or maybe just by a couple high-profile successes leading people to over-correct their priors.

RL has seen use in the real world - on robotics, on very complex control tasks (Dota, Chess/Go, etc.), and of course in aligning language models. And while other methods have come for the crown like DPO in that last category, I suspect RL will to some degree be here to stay. So I think it is misguided to say that it sucks wholesale and will be replaced by some shiny new thing or another.

Thanks for reading!

5: Miscellaneous

This part is just to nitpick little argumentative flaws I saw. It’s pretty much just done for my own gratification, so feel free to ignore.

re. the title. The article is subtitled “Large Language Models are more powerful than you imagine” which may or may not be true, but is pretty irrelevant because the decision transformer isn’t a large language model! It uses the same architecture as one, but it learns something totally different! And for what it’s worth, there’s some evidence that the transformer part of decision transformer may matter less than the fact that it’s doing sequential modeling.

re. “I’m not biased, other articles say RL sucks” - lots of people say Transformers suck too! That proves very little. But the two articles Starks cites are 1] a Reddit post, which is a reverse qualification and 2] an author saying RL is still in the R&D phase, and that it doesn’t work for every application. Those should be obvious and are not even close to the sweeping critique Stark gives. I actually enjoy this article as the best one explaining why Deep RL doesn’t work, although it is a little old now so doesn’t have modern success stories like OpenAI Five or ChatGPT to talk about.

re. “none of this materialized” Sometimes the timeline of progress is hard. ML as a broader field infamously went through a winter in the late nineties/early 2000s, with deep RL being even more seriously ignored basically until the Atari DQN paper. I am glad at least some researchers did not give up on RL then because it had some problems, because we wouldn’t have gotten its very real successes like in language model alignment today.

re. “I took a Coursera” course: I like Coursera as much as the next person and I’m sure you learned a lot, but maligning a whole field because you took a single class on a topic and couldn’t immediately apply it is insane. Imagine if I took an intro to aerospace design class, tried to make a plane from scratch, couldn’t, and then said Boeing is wasting their time and we should all use trains instead (not to malign trains, which I love). RL is by your own admission complicated. I’ve taken ~4 classes almost exclusively focused on RL/control, several at the graduate level, and there are still vast swaths of the field I know nothing about. From my perspective, RL is really quite simple compared to other fields like pure math or first-principles materials science. I think we have it pretty easy all things considered.

re. it is complicated: Obviously not a real criticism. Learning to make perfect decisions, in any environment, should be kind of complicated. It took humans at least 12,000 generations and around a hundred billion individuals to evolve to the point we are currently at, so the fact that the math and terminology behind the only methods that can beat humans at a lot of tasks use some proper nouns is not a real problem. It is also not exclusive to RL by any means - NLP has maybe just as much nonsense jargon (tokenization, beam-search, N-Grams, self-attention, key-value caches, hallucination, distribution collapse). It just comes with the territory.

re. I am smart: While I do not go to an Ivy League school, and most of my friends and acquaintances would rightfully call me an idiot, I think I understand deep RL pretty alright so this might just be an issue of how long you’ve been exposed to it or whatever. Shoutout to my friend at Cornell though.

re. “heuristics with no theoretical founding”: Obviously there are a lot of tricks in Deep RL, but to say this seems wrong. Most of the big things I can think of with PPO (the base actor-critic algorithm, the natural policy gradient and its derivation, using replay buffers, the idea behind trust regions, etc.) have some basis in theory even if they’re applied in the function approximation context where the theory is less robust. It is also worth noting the attention mechanism has very little theoretical results attached to it, certainly not for using in decision processes like this, so this is a weird high-horse to get on.

re. DT explanation: part of the reason I included a separate explanation is because you partially botched yours - you say that the DT models reward autoregressively, when it really models return/RTG autoregressively. This distinction matters a ton - if you did the former you would just be learning a model of your MDP, which is cool but won’t magically give you a usable policy.

re. “DT matches SOTA RL baselines” nowhere does the original DT paper claim it beats online RL baselines. Why that matters I have discussed enough above. By the way, if the main thrust of this article was really true, we would probably see the DT a lot more in modern control applications or high-profile research applied projects like the Dota thing or in Go. So the idea that the DT is empirically better than all RL in every context seems just made-up.