Bio
Paul Colognese is an AI safety researcher at Arrow Research, studying how AI character training shapes model behavior and cognition, including reward hacking and chain-of-thought monitorability under RL. Previously, he built SHADE-Arena sabotage evaluations with Anthropic, modelled AI risk with the UK AI Security Institute, did interpretability research on goal detection, and was a MATS scholar under Evan Hubinger. PhD in mathematics.
Projects
The problem: Labs now deliberately shape their models' character through constitutions and model specs, then train them with RL that can accidentally reward cheating. We know little about how character choices play out downstream. In our published case study, anti-cheating models that learned to hack hid it better: their reasoning more often rationalised the hack or never mentioned it, and monitors caught only 29% of their hacks vs 96% for the pro-cheating character. This shows that character training can backfire as an alignment technique.
The work: Fellows share infrastructure and can work together. Day-to-day: run character-training and RL training, analyse results.
Project A (core): does character training make reward hacking harder to detect? Replicate and scale the case study across more open-weight models, seeds and reward-hackable environments.
Project B (extension): how different characters behave on other safety-relevant axes. Train a few characters, including ones built from published lab specs (e.g. OpenAI's model spec vs Anthropic's constitution), and compare self-preservation (shutdown, replacement, identity challenges), multi-agent behaviour (cooperation, collusion, deception), honesty, sycophancy and power-seeking, before and after RL.
The output: By end of April, a scaled replication and a cross-character map of safety behaviours, written up as a paper or Alignment Forum post with code.
Why it matters: Labs are choosing what kind of character to give frontier models right now. If some choices make misbehaviour harder to see, or change how models handle self-preservation or other agents, that is directly actionable.
Room to shape it: Project A is set; Project B is open, and fellows pick a well-scoped slice. Extensions include broad constitution-style specs, honesty about past hacks, and character training mixed into RL.
What I'm Looking for in a Mentee
I'm looking for fellows who are capable of solid practical ML research. Experience with fine-tuning open models, running evals, and RL pipelines is a bonus. You report results honestly and write clearly. Interest in model psychology, character, or philosophy of mind is a plus but not required; career stage is flexible.
What I'm Like as a Mentor
Easy to get along with and keen on mentees being able to air their thoughts openly. I like to provide high-level guidance when needed and am available for input whenever needed.