CIS 7000 AI Alignment

CIS 7000 Graduate Topics Course

Mathematical Foundations of AI Alignment

Instructor
Aaron Roth
Meetings
TR 12:00–1:29 PM 3401 Walnut, #401B
(Active Learning Classroom)
Dates
August 25–December 7
Communication
Join course Slack

Course outline

Topics

  1. 01

    From feedback to social objectives

    • RLHF as social aggregation
    • Heterogeneous preferences and cycles
    • Arrow’s impossibility theorem
    • Cardinal utility and Harsanyi aggregation
  2. 02

    Preference optimization

    • Scalar reward fitting and DPO
    • Maximal lotteries and no-regret self-play
    • Regularized Nash learning
    • Welfare distortion and identifiability
  3. 03

    Decision-aligned prediction

    • Calibration and decision regret
    • Omniprediction and indistinguishability
    • Blackwell approachability
    • Human–model agreement
  4. 04

    Oversight and assistance

    • AI debate and interactive verification
    • Bayesian persuasion
    • Deference and the off-switch game
    • Cooperative inverse reinforcement learning
  5. 05

    Institutions and competition

    • Best-AI selection
    • Emergent alignment through competition
    • Equilibrium welfare guarantees
    • Evaluation error and assumption audits

Possible extensions: Blackwell’s comparison of experiments; cheap talk without commitment.

Working list

Readings

01

Feedback and social objectives

02

Preference optimization

03

Prediction and agreement

04

Oversight and assistance

05

Institutions and competition

Course materials

Lectures

Research directions

Project ideas

  1. When should an assistant change its understanding of a user?

    People change their minds, make exceptions, and give conflicting signals about what they want. How should an assistant learn from new interactions without either overreacting to every unusual request or clinging to an outdated understanding of the person it serves?

  2. When is another question worth asking?

    Asking for clarification can prevent a costly mistake, but interruptions consume attention and can make an assistant exhausting to use. How should an assistant decide when to ask, when to seek information elsewhere, and when to act on its best understanding?

  3. Can we tell a better answer from a more persuasive one?

    An AI can improve its evaluations by sounding confident, polished, or agreeable without improving the underlying work. How can evaluation reward useful communication while resisting presentation that merely makes an answer look better?

  4. What happens when AI systems teach and judge future AI systems?

    AI-generated feedback could make training and evaluation much cheaper, but it could also pass mistakes and biases from one generation of systems to the next. What determines whether these feedback loops correct errors or make them increasingly entrenched?

  5. Can an assistant learn what we want without learning to steer us?

    The questions, examples, and recommendations an assistant uses to understand someone can also change that person’s preferences. How can we distinguish assistance that helps people discover and pursue their goals from influence that makes them easier for the system to satisfy?

  6. How can we delegate a goal without giving away control?

    Useful delegation requires freedom to act, but a broad instruction should not authorize every action that might advance the goal. How can people retain meaningful control over an assistant without having to supervise every step?

  7. Whose interests does an AI learn, and whose does it leave out?

    The people who provide feedback or pay for a system may be different from those most affected by its decisions. How should AI account for missing voices, unequal influence, and genuine conflicts between the interests of different people?

  8. When should we trust a group of AI systems more than a single one?

    Multiple systems can check one another and contribute different information, but they can also repeat shared mistakes or persuade one another to agree. What distinguishes useful independent confirmation from a larger chorus saying the same wrong thing?

  9. Can short-term feedback teach an AI to serve long-term interests?

    Advice that feels helpful now can undermine learning, reinforce a bad habit, or create dependence, while valuable assistance may be unwelcome in the moment. How can systems learn to support benefits that immediate reactions do not reliably reveal?

  10. When does helping each user make everyone worse off?

    AI agents that negotiate harder, compete more effectively, or capture more attention may serve their individual users while collectively creating waste, conflict, or an arms race. What would allow individually useful agents to produce good outcomes when they interact?

  11. How will AI markets evolve, and who will control the resulting systems?

    Will competition turn powerful AI into an inexpensive, interchangeable service used by millions of independent agents, or will a few companies control both the models and the products built on them? These market structures could create very different concentrations of power, opportunities for competition to provide oversight, and possibilities for auditing consequential uses of AI.

  12. When do “swarms” of AI agents outperform a single agent, and how does this change supervision?

    Dividing work among agents can enable specialization, parallel exploration, and mutual checking, but it can also create coordination failures and obscure responsibility. Which problems benefit from this division of labor, and when does it make human oversight easier or harder?