Abra Ganz & Karl Koch
Projects
Research Direction: When to Speak Up: Early Warning Signals in AI Labs
As labs become more opaque, more information-siloed, and potentially move into the classified realm, there will be fewer and fewer individuals able to spot the early warning signs of AI risk. Employees – insiders – will have unique access to information about lab behaviour, AI systems, and what users’ activity. This project is to assist those insiders in identifying what early warning signs only they can spot. The long-term goal is to help insiders know what they should be on the lookout for and when they should speak up. Specifically, this project is to (i) develop a framework for prioritising what inside information is most critical as early warning signs and (ii) create eye-catching scenarios which represent realistic risks.
More details available here: https://docs.google.com/document/d/1HnO3tCG-oDoB0u5_byroOPYnDflNf8E57zwZJ3b6DTU/edit?usp=sharing
What we’re looking for in a Mentee
Good communication: responds to messages quickly during work hours, pro-actively reaches out if a deadline won’t be met, isn’t afraid to ask clarifying questions or for help & support where needed
Autonomous: we are here to guide you in this research project but also prepare you for doing your own research in the future. As such, an import part of this project will be encouraging you to come up with your own sub questions, ideas for how to answer the questions, and suggestions for how to ensure the policy impact of the project.
What we’re like as Mentors
Each week you will submit the work you’ve done on the project, either as a written document which we will review before meeting, or as a prevention of the directions you’ve explored. We will then have an hour each week to discuss the work you’ve done.
We communicate regularly (via signal) and you should have a low bar for reaching out.
Feedback will be given on how to explore a research question, the best ways to get policy impact, and writing tips. We're very happy to connect you to people in our network to make sure relevant people provide input. We are also happy to support on applications for degrees/fellowships as well as thinking about future career paths.
Bio
Karl has been involved in the AI Safety Ecosystem since ~2016. After years in consulting/ founding and exiting a startup, he founded and now runs The AI Whistleblower Initiative.
Abra has spent her entire career in the AI Safety space, starting in technical research and now working on policy research and advocacy. She is currently Head of AI Policy at Pour Demain and an affiliate at the Oxford AI Governance Initiative.
