Astra Fellowship

Ajeya Cotra leads Open Philanthropy’s grantmaking on technical research that could help to clarify and reduce catastrophic risks from advanced AI. As part of this role, she conducts analysis on threat models (ways that advanced AI could cause catastrophic harm) and technical agendas (technical work that may help to address these threat models). She also co-founded, edits, and writes for Planned Obsolescence, a blog about AI futurism and AI alignment. She is interested in advising technical writers interested in AI alignment and other aspects of AI safety.

Buck Shlegeris is the CTO at Redwood Research, a nonprofit organization that focuses on applied alignment research for artificial intelligence. Previously, he was a researcher at the Machine Intelligence Research Institute. His recent work includes a library for expressing and manipulating tensor computations for neural network interpretability, as well as the papers “Causal Scrubbing,” “Interpretability in the Wild,” “Polysemanticity and Capacity in Neural Networks,” and “Adversarial Training for High-Stakes Reliability.” He is interested in developing evaluation methodologies which AI developers could use to make robust safety cases for their training and deployment plans, examples here, as well as evaluating and improving safety techniques by using these evaluations.

Daniel Kokotajlo has a background in academic philosophy, but now works at OpenAI doing alignment and governance work. In between he did some good work on AI timelines. He's also worked at CLR trying to reduce s-risk and promote cooperative AGI.

Ethan Perez is a Research Scientist at Anthropic, where he leads a team working on developing model organisms of misalignment. He has recently published work on "Discovering Language Model Behaviors with Model-Written Evaluations,” “Measuring Progress on Scalable Oversight for Large Language Models,” and co-founded the Inverse Scaling Prize. Ethan’s research interests include robustness, model transparency, and the development of techniques to better understand and control AI systems. 

Evan Hubinger is a research scientist and team lead at Anthropic leading work on model organisms of misalignment. Before joining Anthropic, Evan was a research fellow at the Machine Intelligence Research Institute. Evan's research focuses on inner alignment and deceptive alignment, concepts Evan and his coauthors coined in the paper "Risks from Learned Optimization in Advanced Machine Learning Systems.” He is interested in work related to model organisms of misalignment, conditioning predictive models, and deceptive alignment.

Fabien Roger is a member of technical staff at Redwood Research. He is interested in finding methods to defend against deceptive models. In particular, he is interested in projects aimed at exploring one of the defenses against potentially deceptive models, such as improving paraphrasing to prevent steganography, training reliable detectors of suspicious activity, or training translators to make models unable to tell if there was a distribution shift.

Hjalmar Wijk is a member of technical staff at ARC Evals. He is interested in work clarifying threat models and creating model evaluations to prevent harm from these threat models. In particular, he would like to make progress on the following questions: What are the most likely ways AI systems trained in the next year or two could cause catastrophic harms? What are the capabilities that enable these threat models, and how (at a high level) could we evaluate for them? At what levels do these risks clearly demand strong containment measures like state-proof security? There is a lot of conceptual work involved with mapping out this space, understanding future AI capabilities and identifying key considerations, but there is also an opportunity for more concrete work talking to diverse experts, identifying current bottlenecks to harm and finding reference classes or historical examples.

Lukas Finnveden is a member of Open Philanthropy's worldview investigation team. He is interested in issues other than intent alignment that might be important if transformative AI happens in the next decade or two, such as: how society should treat digital mind as it becomes increasingly plausible that they are sentient, ways to increase the probability that humanity gets on a good deliberative track to eventually reach good empirical & philosophical beliefs, and ways evidential cooperation in large worlds could be decision-relevant.

Megan Kinniment is a member of technical staff at ARC Evals who leads a variety of research projects. ARC Evals’ current research focuses on eliciting and measuring capabilities of frontier models, as well as threat modeling work to understand the minimum capabilities needed for autonomous AI to pose a catastrophic risk. They are interested in projects relating to LLM agent development, task creation, threat modeling, and model assessment.

Owain Evans leads a research group in Berkeley and has mentored 25+ alignment researchers in the past, primarily at Oxford’s Future of Humanity Institute. He is currently focused on defining and evaluating situational awareness in LLMs (relevant paper), learning how to predict the emergence of other dangerous capabilities empirically (see “out-of-context” reasoning and the Reversal Curse), and honesty, lying, truthfulness and introspection in LLMs. 

Richard Ngo is a member of the OpenAI Governance team. He was previously a research engineer on the AGI safety team at DeepMind and was one of the main developers of the AI Safety Fundamentals curriculum. He is interested in AI threat modeling – understanding both why AIs might be misaligned and what misaligned AIs might do – as well as understanding defensive uses of AI and forecasting the far future. 

Rob Long is a Research Associate at the Center for AI Safety, where he works on issues at the intersection of philosophy of mind, cognitive science, and ethics of AI. Ethan Perez is a Research Scientist at Anthropic, where he leads a team working on developing model organisms of misalignment. They are interested in finding methods to determine whether AI systems possess consciousness, desires, or other states of moral significance. Self-reports, or an AI system’s statements about its own internal states, could provide a promising method for investigating this question. They are interested in testing whether current language models can learn to introspect in ways that generalize to other introspective questions, a necessary condition for eliciting reliable AI self-reports. 

Tom Davidson is senior research analyst at Open Philanthropy. He works on assessing whether transformative AI might be developed relatively soon, how suddenly it might be developed, and what impact it might have. See his recent report on AI takeoff speeds. He is interested in various projects related to this work: in particular, whether the biggest AI algorithmic discoveries of the last ten years would have been possible without fast-growing amounts of available compute, and how important Responsible Scaling Policies might be for reducing risks from accelerating AI progress.