Bio
I'm a research lead at Arcadia Impact's alignment team.
Interested in doing ambitious alignment science, aimed at supporting a pause
Projects
Research Direction: Science of model motivations + alignment
Labs increasingly include alignment-focused environments as part of RL training. Anthropic mentions training on reward-hacking honeypots; OpenAI describes 'beneficial RL'. However, the details of this training remain extremely unclear. The goal of the project will be to attempt an informed open-source replication of alignment RL and gather evidence about how well it works.
A basic result would look like: (i) implementing a realistic RL pipeline (infra + env mix), (ii) doing training run(s) at scale, (iii) studying the alignment propensities of the resulting trained models. A stronger work would extend this by doing detailed ablations to understand the factor(s) that determine the effectiveness of alignment RL.
This will give us evidence about: how well does alignment RL work? Does it compensate for misaligned RL environments?
One possible outcome is that RL predictably leads to misalignment; if so then we might want to demand regulators ban RL. Rigorous high-quality evidence is necessary in order to support this claim.
This describes one out of several projects I am interested in; I am open to other projects in the space of "science of RL motivations". Examples of things I'm thinking about include metagaming and motivated reasoning.
What I'm looking for in a Mentee
Strong fellows will be capable of quickly working on tasks + resolving low-level blockers. Strong fellows will take substantial ownership of the project direction - coming up with suggestions for what they'd like to do, noticing interesting things and surfacing them to me, and keeping in mind the high-level theory of change.
What I'm Like as a Mentor
I like to have a high volume of communication with mentees - spam me with Slack messages / Gdocs
I'm pretty hands off by default. I think mentees should drive project(s) and develop their own judgment.
I like meetings to be pretty focused - come prepared with an agenda and focus on the most important points
Mentees should proactively ask me a high volume of questions if they are unsure about anything - whatever's needed to unblock.