Reflections on RLC 2025

This August I was lucky enough to go the second Reinforcement Learning Conference (RLC) which was in Alberta, Edmonton, Canada. I wasn’t presenting any work there (I am a bit of a bum and don’t even work in RL) but my advisor was gracious enough to send me and one of my lab mates so we could get a sense of what the RL community has been thinking about.

Here are some miscellaneous impressions in no particular order:

1. Edmonton is fine. I hate to be an obnoxious Toronto elitest, but Edmonton is really quite out of the way even for the prairies and the city was unremarkable. It was hosted at the University of Alberta though, which has a quite nice campus. For those interested in the history, the University of Toronto (where I go) was modelled architecturally and organizationally off of the old British Oxbridge system, so much more buildings in the gothic revival style and a bunch of separate constituent colleges. It’s also a fairly small campus because it’s embedded fairly snugly downtown. UofA took inspiration more from American land grand universities so it has a sprawling, more well-defined campus with classic American university things like a central quad. RLC was at UofA this year because it was at the University of Massachusetts Amherst last year, and both schools were super pivotal to founding RL because Andrew Barto is a professor at UMass and Rich Sutton was his student, who then later moved to UofA and started a large RL community there. But in any case I like the sort of aesthetic contrast between UofT and UofA in their British/Americanness which nicely reflects Canada’s mixed heritage.

2. Related to the above, the presence from the Alberta and UMass folk was very strong and presence from everyone else was less strong. This was unsurprising to me: in my mind UofA/UMass are very central to the RL “cult” where they have a very high density of connected researchers working on fairly pure RL development. People doing more applied stuff or focusing heavily on niche domains (like Cal’s offline RL groups, MIT’s diffusion for RL people) spend their time more in the robot learning crowd. But because both are so big in RL, the star power draw was just remarkable and the prestige of the keynote speakers was matched by their quality.

3. On RLC’s “philosophy” as a conference: it is much more focused on work that might not be advancing SOTA or benchmark-chasing, but is robustly theoretically or experimentally supported. This means a lot less papers of the vain “we make this small asinine trick/change to an algorithm and it gets 5% better at Atari” and a lot more “we study this particular interesting sub-problem on toy environments”. I think generally Neurips/ICLR/ICML are more in the former’s vain where to convince reviewers you need some sort of improvement on a standardized benchmark against a reasonably-picked set of baselines. I think I used to be much more bullish on the RLC approach, but I think after going I’m now a bit more in the middle because seeing paper #8 that only experiments on MuJoCo control environments or whatever made some of the arguments some of the papers were making much less convincing in the sense that I don’t get a very strong signal of if their findings would apply universally. I will admit this is a very hypocritical view given I work on toy domains too, but I digress. Ultimately I think having both kinds of venues is probably quite healthy for the field though.

4. Things I was surprised there was a lot of work in: feature learning (especially in the successor/forwards-backwards area), multi-agent RL. I was surprised about the feature learning thing because I never got the impression the benefits of these methods were particularly large: when I talked to my lab mate about this he seemed under the impression the benefits are big for non-reward unsupervised offline RL where I might want to specify some arbitrary task/goal to reach and these methods let you do that zero-shot, but when I pressed him on the empirical claims to back that up he seemed to think current methods don’t get there yet. I was also surprised by MARL I think just because my lab is mostly focused on manipulation so I never run into any multi-agent works.

5. Things I was surprised by the absence of: generative models for RL, offline RL, hierarchical RL. The first might be because I work in this so I naturally suspected more, but I think there were like two papers about diffusion-based policy learning and none about diffusion planning (what I currently work in). This seems weird because there’s I think a nicely amassing body of work showing that diffusion planning is quite practical. Offline RL was also quite weird: again, biased because most generative RL work are in the offline setting and my lab mate works on this so I also think about it quite a bit, but there wasn’t a ton here. Hierarchical RL was maybe the most surprising though and I’ll talk more about why later.

6. Things that had the amount of work I would have suspected: RL for LLMs (although not a ton of reasoning stuff weirdly), algorithmic work, theory developments, architecture work, meta learning. Not a lot to say about this.

7. What did I change my mind about? Interestingly, my lab mate and I disagreed on this: they came out thinking feature learning was the next big thing and I thought hierarchy was. I’ll discuss my view first. Hierarchy in decision making was already something I’ve been thinking a lot about as I realize generative models are actually not particularly great at handling low-level control in stochastic environments (ie, generating a really long horizon set of joint torques and executing it closed-loop doesn’t work well compared to a strong control policy) but it does work much better for high-level planning because you bypass a lot of the struggles RL has with propagating reward through a long time horizon. I thought two of the keynote talks heavily validated my view. The first was Leslie Kaelbling’s talk, which was sort of more about robotic planning than RL and discussed various ways of decomposing robot problems into sub-tasks you can apply learned models to. So for instance do perception with one module that handles object recognition, handle planning with a symbolic AI thing or with some other learned approach, etc. A big message of her talk was that even though her proposed methods don’t use RL in the traditional sense (no TD learning, Q-Functions, policy gradients, etc. except maybe to learn short horizon controllers) they did have the holy grail RL properties (ability to learn from experience alone, ability to continually learn and adapt, etc.) and one of her big things was that an explicit control/planning decomposition works exceptionally well for robotics, which is true. The second was Richard Sutton’s talk, where he proposed a model for developing superintelligence he calls OaK which stands for Options and Knowledge. That hierarchy is important in his model comes from the “options” part which is one of the traditional ways of formulating Hierarchical RL. Another talk that came to similar themes was Dale Shuulmann’s, where he talked about kinds of problems that LLMs can/can’t solve and one of the big takeaways was that for some kinds of problems LLMs need to explicitly spend time planning symbolically at a high level of abstraction, which gets much more expensive if you can only do such planning in a flat hierarchy where reward needs to get propagated through so much further.

So why has RL seemingly abandoned hierarchical RL when everyone agrees its so important? I do not know to be honest, I’m not very deep in the literature, but I suspect it’s a combination of a] its hard because of weird learning dynamics when you try to learn options b] if you are an engineer its usually fairly easy to manually design your options/primitives in which case there is no reason to bother with learning them, so not a lot of works that do have implicit hierarchy actually develop new hierarchical RL methodology.

1. Last thought on academic stuff- it seems to me like there is a growing disconnect between the RL and robot learning communities, which have traditionally had an on and off relationship. My rough understanding is that until ~2004 there wasn’t a ton of interaction because RL was too primitive to use for much besides toy tasks (this was because the right combination of variance reduction tricks hadn’t been developed so you were mostly stuck with hard-coded features and linear Q-Learning/SARSA/PG), so RL people did their tasks in gridworlds/games and robot learning people used classical control methods for their robots. Then early stuff like Andrew Ng’s thesis about RL for drones, Sergey Levine and Chelsea Finn’s work in the late 2000s/early 2010s about RL on robots came out and people realized there might be something here so the communities became closer. Deep learning finally starts to work in ~2012 with AlexNet, which revives deep RL. DQN comes out in 2013, policy gradient hits it big with PPO in 2017 and John Schulmann’s work at Cal about RL for humanoid robotics makes the communities even closer. The Cal and MIT people then do even more through the early 2020s including applying RL to real robots and it seems like they are very closely related. When I applied for PhD programs in 2023 I largely applied to robotics labs because so many of them were doing RL of some kind. But I think a disconnect started in maybe around 2023 with the Diffusion Policy paper leading to VLAs which showed imitation learning scales very nicely for robotics, and now we are in the modern age where robot learning people it seems like are mostly focused on different ways of doing generative imitation learning. At RLC it didn’t feel like there were many robot learning people really, I don’t think I saw more than maybe one or two papers out of ~100 main proceedings and ~150 workshop papers that had experiments involving real robots, or even realistic robot settings in simulation. It very much felt like RL people had stopped thinking “how am I gonna apply my method to a robot” and are hanging up the roboticist cap and moving back to the hobbit-hole of their own particular subfield, happy to tend the garden of more general purpose methods until the robotics people hit a wall with the BC based approaches or a problem better suited for RL comes along for the community to fixate on again.

2. Was anything about the trip fun? I take a while to warm up in social situations, even amongst my peers and even in comparison to other socially handicapped computer science grad students, but I did find I was able to have some great conversations with people about their work after I got used to being in a conference setting on like day 3ish. Won’t name anyone specifically because that feels weird, but if you remembered chatting with me I want to extend my gratitude because I didn’t have a single bad conversation at the conference. The banquet was absolutely awesome, especially the improv comedy group that worked a lot of the RL profs into their routine. I also visited the West Edmonton Mall, which made me a bit embarrassed as an American that such a consumerist monstrosity (the mall has a theme park, water park, several mini golf courses, and the largest parking lot on the planet) was in Canada and not Ohio or something.

Thanks to UofA, the conference organizers, the sponsors, et cetera et cetera. RLC will be in Montreal next year, so there’s a slim chance I will be able to give more thoughts then.