Bio
Alexandra is a core maintainer of Inspect AI and its ecosystem packages, among the leading open-source frameworks for AI evaluations and research. She has spent several years working on AI evaluations and the science of evaluation, designing and developing evaluations while leading engineering teams and coordinating research and engineering programmes. She has worked closely with the UK AI Safety Institute (UK AISI), most recently co-authoring Transect, a tool for analysing agent transcripts to better understand long-horizon agent behaviour. She is also a member of the European Commission’s Scientific Panel on AI.
Projects
Project 1: Auditing AI evaluations for regulatory use
How should regulators use AI evaluations as evidence for enforcement? This project will develop a framework to help regulators, particularly in the EU, assess what an evaluation measures, how reliable its results are, and how to interpret them in context. The work will synthesise existing research and frameworks, develop and test an auditing methodology through real-world case studies, and explore what additional evidence is needed to contextualise evaluation results. By April, a strong outcome would be a research paper and practical report with a comprehensive framework and worked examples, potentially accompanied by AI-assisted auditing tools. Misinterpreting evaluation results could lead to critical risks being overlooked.
Project 2: Measuring the reliability of AI evaluation scanners and monitors
Scanners analyse AI agent trajectories post-hoc, while monitors analyse agent behaviour live during evaluations, helping identify suspicious behaviour, evaluation awareness, and other patterns. But these tools can themselves be unreliable: their judgements may be inconsistent, or they may fail to recognise problematic behaviour. Building on existing work on scanner reliability, this project will develop statistical methods for measuring and improving the trustworthiness of scanners and monitors, potentially through repeated sampling and aggregation. The goal is a research paper, experiments on real use cases, and potentially an implementation in Inspect AI, with clear guidance on how users should interpret reliability metrics.
Project 3: Understanding multi-agent AI behaviour through transcripts
As AI systems increasingly operate in teams, how can we understand the complex dynamics between agents? This project will develop methods and best practices for analysing multi-agent trajectories, including useful analytical questions, visualisations, and data structures for capturing agent interactions. Work may involve generating datasets, experimenting with transcript-analysis techniques, and contributing to Inspect AI's support for multi-agent systems. A strong outcome would be a paper and potentially open-source tooling that makes otherwise opaque multi-agent behaviour easier to investigate.
All three projects are flexible: fellows can refine the question, methodology, and deliverables to suit their strengths and interests.
What I'm Looking for in a Mentee
I prefer fellows to be self-directed and take ownership of their projects.
For the regulatory evaluation auditing project (1), a background or strong interest in technical AI governance would be valuable. A strong report could be shared with users in the EU AI Office, so familiarity with this audience would be valuable.
For the scanner reliability (2) and multi-agent analysis (3) projects, I'm looking for fellows with strong research engineering, machine learning, or statistical backgrounds who enjoy a mix of conceptual work and implementation. Mid-career professionals are preferred.
All three projects would require good writing skills.
What I'm Like as a Mentor
I'm a hands-on research engineer who works on code daily, so I'm happy to dive into technical details. I travel frequently, so I prefer short, informal check-ins over Slack, supplemented by remote calls. I'm based in London and work from LISA, so I'm also happy to meet in person.