My Top 5* Favorite Papers

Recently (well, like a month and a half ago now) for a grad school interview a professor asked me to send them my 5 favorite papers, with some explanation for each. I had a lot of fun thinking about and writing the descriptions, so I figured I would put it somewhere permanent.

In no particular order…

1] Mnih, Volodymyr, et al. “Playing atari with deep reinforcement learning.” arXiv preprint arXiv:1312.5602 (2013). This paper I think requires no introduction, it was to my knowledge one of the first high-profile successes in Deep RL and a foundational work. Whenever I implement DQN from scratch I go back to the original paper to use as reference (instead of an existing repo or something) because its explanation of the algorithm is very sound. I found the way the DQN paper writes it by decoupling the original Q-value estimate and new target, then writing those as MSE loss inputs helped me understand what the first-order optimizer is actually doing, and why using another optimizer besides SGD like RMSProp in their case isn’t totally ill-informed. I put the original 2013 preprint instead of the 2015 Nature paper because the preprint highlights the algorithm more prominently which I think is important to understanding how this method works, but one nice thing the Nature paper does is show t-SNE embeddings of the neural network. This was to demonstrate (I think) that the network learns useful representations just from the Bellman update, without any intermediate representational learning objectives which I think is part of what makes Deep RL truly remarkable: that you can learn what from your data is important to pay attention to, just from (parametrized) TD learning!

As an aside, I find it amusing that when early methods for “understanding” neural network behavior were invented circa 2014, a lot of papers shoehorned them in (a paper I greatly enjoy but that is not on this list, Learning to Fly by Driving, does this as well with activation map analysis) but that seems to have more or less completely gone out of fashion.

2] Agarwal, Rishabh, et al. “Deep reinforcement learning at the edge of the statistical precipice.” Advances in neural information processing systems 34 (2021): 29304-29320. This paper investigates some of why the Deep RL community has such trouble with reproducibility and uncertainty and proposes some new methodological changes for how RL agent performance is reported. I think this work does a strong job at isolating and separating two different problems: that of the unreliability of RL agents generally, and how the community reports their findings. That RL agents often fail, even SOTA models/algorithms on relatively simple tasks, to converge some significant percentage of the time indicates that RL is just really hard, which isn’t the RL community’s fault: learning optimal control in an unknown, gigantic MDP is an incredibly difficult problem. But this paper identifies that the way results are just reported leaves room for improvement. As I became personally familiar with how hard RL is compared to supervised learning I gained a deeper appreciation for why good reporting matters and keep this work in mind whenever I’m working on reporting very noisy or expensive performance metrics for any task.

[Another note - I have noticed recent examples of the RL community continuing to commit small amounts of experiment hackery. I’m not going to call out the specific paper I am talking about, but they essentially compared their form of recurrent network for RL to a FFN on a POMDP, instead of a different kind of recurrent network. Of course your network will work better in that environment!]

3] Schwarzer, Max, et al. “Bigger, Better, Faster: Human-level Atari with human-level efficiency.” International Conference on Machine Learning. PMLR, 2023. This paper claims to have achieved superhuman mean IQM performance on the Atari 100k benchmark (a sample-efficiency benchmark where an agent has 2 hours of access to a set of the Atari environments to learn) for model-free RL. I like this paper because in my view, there are maybe ~4 big picture things preventing Deep RL from really making a big impact: it takes forever to do, exploration is unsafe in the real world, it doesn’t converge a lot of the time, and dealing with sparse rewards/reward shaping is hard (of course, these are all related, and I could be wrong about all of this). I wish I have read more in the RL safety and robustness literature! But I have read a bit about sample-efficient RL, both deep and theoretical, and I think this work’s approach is quite cool. The gist of what it does is that it combines 1] an intermediate supervised next-state prediction objective to help the model learn useful representations faster 2] all the classic DQN tricks that Rainbow showed help sample efficiency, but especially smarter exploration 3] frequently resetting parts of the network to a random baseline, and keeping a massive replay buffer to help the model more quickly adapt to new, better Q-Function estimates. I am actually not sure that this is really “model-free” RL because the intermediate supervised objective is doing next-state prediction, which to me feels like sneaking in model RL on a technicality even if it is just to learn good representations faster and not to do MPC or something. But regardless I think that combining these is very intuitive: in the real world we seem to both try to guess what will happen if we take actions and update our knowledge of how good actions are after the fact, and the fact that it seems to work well is very nice.

[Note - I recently had the pleasure of listening to a talk at BU about good RL representational learning which was absolutely fascinating, and approached this problem from both a theoretical and practical perspective. The presenter was Dr. Xuezhou Zhang, if you are interested in looking more into his work]

4] Wang, Guanzhi, et al. “Voyager: An open-ended embodied agent with large language models.” arXiv preprint arXiv:2305.16291 (2023). This is a paper that uses GPT-4 to create an embedded agent that can learn to play Minecraft reasonably well. This introduced me to the question of why LLMs in RL are interesting: can we use the general knowledge and reasoning capabilities of LLMs to accelerate RL learning? The idea of “transfer learning” general knowledge about the world to robotics/control tasks seems pretty important. If I was dropped into a field in a country I had not been to (say, Thailand) and told to move 2 miles without hurting myself, the fact that I generally know how to walk, know what trees/buildings/other obstacles are, am aware of dangerous animals, and have some biological preconditioning all means I don’t need to spend 5 trillion days trying and failing to walk around Thailand (like RL agents do) before I can learn safe Thailand navigation, even without prior familiarity of Thailand’s traffic laws or local fauna. The fact that RL agent performance on Minecraft, a pretty approachable videogame for humans, is so poor speaks to some of those problems I talked about in the paragraph above. But I think a big component of the sample-efficency and safety components will be about transfer learning and incorporating general knowledge into RL agents better. This work really opens up the door for so many other ways you could integrate LLMs (multimodal with vision, presumably) as a form of this “general knowledge to accelerate RL” idea: using them to generate reward/advantage signals, or to do smarter exploration, or to handle low-level tasks (like for Minecraft, using an LLM to write a script for mining, which the RL agent can use as an action instead of mouse/keyboard commands) which gives the RL algorithm itself a more limited task to learn. It would be cool if there was some more general learning framework to apply to learn knowledge that transfers between control tasks (rather than using language as an “intermediary), which seems like an open and very difficult task. If anything my main problem with this work is that Minecraft seems like too well-known of a thing to expect this method to generalize well to other environments for, and that it requires direct Minecraft API access.

5] Silver, David, et al. “Reward is enough.” Artificial Intelligence 299 (2021): 103535. I originally got into the broad area of AI safety from my Philosophy background, and this paper was part of what led me to want to study RL more deeply. A lot of my thoughts on why I think RL is such a powerful framework are in my SOP, so I won’t restate them here, but this work deepened my belief that the foundations of RL, machine intelligence broadly, and the philosophy of intelligence are intimately related. For instance, there is a lot of buzz around if LLMs are “conscious” or a pathway to AGI. I think that language modeling on its own is not sufficient because that will always restrict the model to the distribution of human language use, which is a limiting factor because that distribution changes as people invent new things for various reasons. But this work makes an argument on language similar to philosopher Ludwig Wittgenstein’s eventual position on language, which is that viewing it as a formal rule-based system is less meaningful than as a tool for people to get what they want. Humans do not use language because they are trying to replicate a distribution or follow grammar rules, but to communicate ideas, persuade people of things, et cetera. I love the fact that this paper came out before it was discovered that LLM toxicity issues can partially be alleviated by using a reward structure in its training through RLHF because it is a perfect example of a well-founded, philosophical principle bearing out in the practical world. This paper is basically a well-researched opinion piece, and there are parts of it I think are not well justified (for example, I don’t see how reward by itself is enough for generalization – there are lots of poor transfer results in RL that seem to strongly disprove this) but it helped guide my thinking both from a practical comp sci/engineering perspective and a bigger picture one, so it is one of my favorites.

* BONUS

Schoenig, Richard. “The free will theodicy.” Religious studies 34.4 (1998): 457-470.

Because I applied for PhD programs in CS (and related areas) I didn’t include any of my favorite philosophy papers (if you did not know, I am a philosophy major). But this one is excellent.

For context, I wrote a very short paper talking about the history of the Ontological Argument (OA) for a class on early modern philosophy. Put very briefly, the OA says that because God is by definition the greatest conceivable being, and because the property of existence is a more good property for a good thing than non-existence (simple example - a real ice cream cone is in some way “better” than one you’re just imagining about - you can actually eat the one in reality!), then God must exist because he does by definition. To normal, well-adjusted people this seems pretty stupid but the deeper you get into the argument, especially the very obvious objections (you snuck in God’s existence as a premise, you are just saying a tautology, how is existence a property, why is that property good, etc.) the more it becomes clear it is a very challenging paradox. That paper is accessible on this website if you want to know more.

Getting back on topic, a very common argument against God’s existence (that sort of has nothing to do with the OA, but writing that paper led me to it so it’s related in a sense) is that evil seems to exist in the world, and an all-powerful and all-great Abrahamic God would have no reason to let that happen. This is called the problem of evil, and solutions to it from theists are called theodicies. (For what it’s worth, I think the problem of evil is more or less a mercy killing against the Abrahamic interpretation of God. I’ve never heard an explanation of why evil exists that doesn’t either admit he’s not all good or all powerful. But frankly I think the interpretation that God is just a jerk sometimes is consistent textually with the Torah/Old Testament/Qu’aran, so take that for what you will.)

A common theodicy is called the free-will objection, which basically says that if God forced the world to be perfect it would violate people’s free will to do evil which is in some fashion or another a kind of evil even more heinous than the evil people commit themselves. A very quick Google search says a guy named Alvin Platinga either came up with or popularized this argument. Dr. Platinga, it should be made clear, is a brilliant theist philosopher - in my OA paper I referenced a lot of his work extending the OA to modal logic which is a very cool and fun idea because it basically says God only has to exist in one possible universe, and then he can travel between dimensions to get to ours.

But getting back to the paper, this paper critiques the free will theodicy and does it in a way that is sheer elegance. I help with editorial work for BU’s undergraduate philosophy journal, and so I read sort of a lot of argumentative philosophy essays. I think this is the strongest I’ve read; it is incredibly clear, well structured without feeling lumbering, and just makes some fantastic arguments. It is the kind of singular vision, one person on an argumentative mission to annihilate somebody else’s position piece that is common, but not commonly very good. This one is better than very good (in my opinion - I am pretty dense).