Research Assistant

After my freshman year internship I knew that reinforcement learning was the area I wanted to go deeper on, and that the fastest way to do that was to see the algorithms and training pipelines being applied to real problems. During my sophomore summer I reached out to Professor Raman Arora and joined his group's work on Tanks World, a multi agent battle arena environment. I was working on other things over the same summer, so this was a part time role rather than a full time one.

The research question was about convergence. In single agent reinforcement learning the goal is to find the action that maximizes expected return, and in a stationary environment there are algorithms with real convergence guarantees, at least under simplifying assumptions, and the deep learning versions of those algorithms work well in practice. In a multi agent environment the other agents are part of the environment, and if they are learning at the same time as you, the dynamics shift on every iteration and the optimal action shifts with them. You are sampling from a distribution that changes underneath you. The question the group was asking was experimental rather than theoretical: if single agent algorithms such as PPO, TRPO and REINFORCE are dropped into a distributed learning setup where every agent is learning in the same environment, do they still converge to a useful policy, and does any interesting emergent behavior show up along the way? Nobody expected an optimal policy, just a useful one.

Running the experiments, and the training pipeline they ran on. The pipeline had originally been written by the Johns Hopkins Applied Physics Laboratory, and it was my first exposure to distributed learning, so the codebase was a lot to take in at first.

The documentation was thin and the code needed work, so I asked Professor Arora whether I could spend part of the summer refactoring and documenting it. The new experiments and better model visualization also needed changes to the training pipeline itself. He agreed, so the role became a mix of running experiments with different reward functions and training configurations, refactoring and documenting the pipeline, and writing new algorithm implementations in PyTorch, since most of the existing code was in TensorFlow.

I treated the summer as an opportunity to learn as much as possible, which meant reading and documenting the codebase before I could change it with any confidence. From there the work was building out an experimental grid. The axes were the reward function, whether the tanks shared a single policy or each learned their own, distributed training against centralized training, and which algorithm each agent ran, including matchups where tanks running different algorithms played against each other, to see whether that variety helped all of them improve. Each scenario was coded out by hand and run on the group's compute cluster.

The documentation and the refactored pipeline, which made the next round of experiments easier to set up than the first one had been, and the PyTorch implementations sitting next to the original TensorFlow code. For me, the lasting part was learning how distributed training actually runs: MPI, batching work across a cluster, and setting up and running experiments on shared compute.

I would simplify. Research code that sits on top of a lot of theoretical math is easy to get lost in, and I spent time digging into the internals of the APL pipeline that I did not need to understand yet. Knowing what I know now, I would rely on the abstraction they had already built, assume their functions work, and only open them up when a bug forced me to. Writing down the task and staying focused on exactly what I was asked to do, before going above and beyond, would have given a better return on the time.

The other change is asking for help sooner. When an algorithm or a training paradigm was unfamiliar I tended to work through it alone, and in a research setting, where the people around you have already solved that exact problem, that is the slow way to do it.