Funded Grants

Dissecting learning: combining experimental and computational approaches

A longstanding question in psychology concerns the representational content of learning: what information is learned from experience? Consider a trial-and-error decision task such as an animal foraging for food or a human practicing chess. An early behaviorist proposal, due to Thorndike, was that such learning is limited to action propensities. On this view, successful actions (those followed by reward) are stamped in or reinforced as stimulus-response habits, and more likely to be repeated later. The refutation of this narrow view of action animated the cognitive revolution. Tolman, for instance, argued that organisms are not doomed merely to repeat previously successful behaviors, but can learn about the structure of a task (a "cognitive map" of a maze- the strategy favored by a chess opponent) and draw on this knowledge to evaluate even novel actions: new routes or moves.

Surprisingly, this debate has not been resolved -- indeed, there is evidence that both sorts of mechanism coexist -- and the same ideas suffuse contemporary thinking. In neuroscience, Thorndike's notion of reinforcement stamping in habits persists in prominent computational hypotheses of the action of the neurochemical dopamine in striatum. This same reinforcement mechanism is widely invoked in relation to the compulsive effects of drugs of abuse, which affect dopamine. Much less is understood at either process or neural levels about the nature of more cognitive learning: what precisely is represented, how this is learned and influences decisions, and how these, more deliberative choices interact with more automatic (and in some cases, potentially damaging) habits.

In my lab, we are making fresh progress on these venerable questions by leveraging two technical innovations. First, new computational theories clarify the psychological mechanisms. My doctoral training is in computer science, where research in reinforcement learning has developed a range of approaches to trial-and-error learning about decisions. This area offers a taxonomy of precise hypotheses for how the brain could approach this problem. In the last decade, one class of reinforcement learning algorithm has had dramatic success explaining the activity of dopamine neurons and their putative action driving habit learning. In an influential theoretical article, we proposed that a second class of algorithms, called model-based reinforcement learning, could provide a similarly quantitative account of the more poorly understood cognitive or deliberative, map-based learning strategy.

Second, computational characterizations of these computations enable newly quantitative experimental investigations of them. Classic approaches study learning curves to examine net changes over many trials. Computational algorithms, in contrast, are trial-by-trial hypotheses about how decision preferences should adapt following each trial's feedback. They may be tested directly for trial-by-trial fit to raw choice sequences, measuring (for instance) the differential contribution of different sorts of learning. Similarly, they provide dynamic predictions about otherwise subjective quantities hypothesized to underlie the decisions -- for instance, subjects' expectations about rewards -- which can analogously be tested against decision-related neural signals.

Together, these new hypotheses and new ways of testing them have delivered a rich picture of habit learning. Future work in my lab focuses on extending these techniques toward a similarly detailed understanding of more cognitive decision learning, and how this interacts with habits.