This workshop is part of the How to Think about Tech? The Case of 'AI Safety' study group initiated by some of the fellow candidates of the 2025/2026 Introduction to Political Technology course. It is open to faculty and fellowship candidates only - external guests by request (please contact us explaining your interest and area of expertise).

AI companies release new models constantly, often claiming each new one is "safer" than the last. But how do they actually know? What tests do they run? What gets measured and evaluated?

This workshop provides a technical primer on AI evaluations (evals) - the practice of testing AI systems for capabilities, safety, and fairness. We'll explore:

What evaluations are and why they matter for AI governance
Different types of evaluations: capability, alignment, safety, and societal impact assessments
How evals work technically: constructing datasets, defining metrics, and designing evaluation protocols
Examples that demonstrate the mechanics in practice
(Optional) Hands-on activity: designing evaluations for a specific scenario to surface the tradeoffs between what we can measure and what actually matters.
Discussion

This session is designed as a collaborative technical primer. I've been studying and working with evals over the past year as part of an ML curriculum, and I’m excited to share what I’ve learned. If you have deep experience in this area, please reach out as we'd love to learn from you!

This workshop pairs with the AI Safety study group discussion on Feb 22 on AI Evals (https://luma.com/357o5glx) examining the governance and political implications of evaluation frameworks.

AI Evaluations - a technical primer workshop