Tuesday, April 14, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

Want to know what works in teaching?

Wednesday, April 8, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

Want to know what works in teaching?

Monday, March 2, 2015
Winnett Lounge – Winnett Student Center

The Secret Life of a Snowflake

How the Brain Learns from the Past and Makes Good Decisions for the Future: A Tour of Neural Reinforcement Learning

Watson Lecture Preview

It is often said that people who do not learn from history are doomed to repeat it. Not being one of those people requires a network of different brain regions to work in concert. On Wednesday, February 4 at 8 p.m. in Caltech's Beckman Auditorium, John P. O'Doherty, professor of psychology and director of the Caltech Brain Imaging Center, will discuss our current understanding of how we learn from experience. Admission is free.


Q: What do you do?

A: I study how we learn from experience. Humans and other animals have to make decisions all the time to maximize their benefits and minimize danger. These decisions range from what I should have for dinner or should I cross the road—which could have life-changing consequences if I'm wrong—to the selection of a life partner. I don't claim that "Who should I marry?" is equivalent to "Carrots or Brussels sprouts?" but we do think that many decisions share certain commonalities. So we look at very simple tasks that give us a window into how the brain solves problems to maximize future rewards.

We study brain activity by putting your head in an fMRI scanner. "MRI" stands for magnetic resonance imaging, and you've probably had one if you've had a sports injury. The "f" stands for "functional," and an fMRI scan detects changes in the oxygenation levels in the blood. If a certain part of the brain is active, its oxygen supply increases. We map those increases onto the brain's anatomy in 3-D while our volunteers perform some task that involves learning.

A task might be playing virtual slot machines. You have a choice of three machines, and we tell you one machine pays better than the others. So you choose one, press the button, and get instant feedback—you win or you lose. As you try to work out which machine is better, we monitor the patterns of activity in various parts of your brain. One of our goals is to find the part of the brain that represents the experienced value of the things we meet in the world—how good it feels to win, or how bad to lose.

We're also interested in how the brain changes its expectations. As you play the machines, you're constantly revising your estimate of which machine is better. We have computational models that we think represent how the brain internalizes feedback, and we're trying to find brain areas where the activity matches those models.

We think that understanding the neural circuits and computations that underpin our decision-making capacity may shed some light on certain psychiatric disorders, such as obsessive-compulsive disorder, depression, and addiction. On some level, all of these can be seen as decision-making gone wrong. Addiction, for example, involves a choice—voluntary or otherwise—to engage in a certain pattern of behavior.


Q: Setting aside clinical disorders, why do people make garden-variety bad decisions? What leads us to cross a busy road and almost not make it?

A: First, it's important to emphasize that humans are collectively pretty good at making decisions. That's why we've been so successful as a species. But there could be all sorts of reasons why an individual might make a poor decision. For example, you might underestimate how fast the traffic is moving.

My lab is particularly interested in how two distinct decision-making mechanisms may interact to produce bad outcomes. One mechanism is "goal-directed," in which you evaluate the consequences of your action in light of the goal you're pursuing. This requires a lot of mental energy. In contrast, "habit-controlled" decision-making is basically stimulus-response—you react to some cue from the environment. Habits can be very beneficial, because you can execute them quickly without thinking deeply. Once you learn to ride a bicycle, for example, you don't have to concentrate on keeping your balance. It becomes routine, and you can focus your mental energy on other things. Poor decisions can result when the habit system drives your behavior when you really should be solving things in a goal-directed manner. This may be how addiction becomes compulsive. The goal-directed system says, "I don't want to take this drug any more," but the habitual system overrides it.


Q: How did you get into this line of work?

A: Even as a kid I was interested in science and its unsolved mysteries. I was actually keen on astronomy as a teenager and really considered going in that direction. Then I started getting interested in how computers work, which led me to start wondering about how the most complex computer that we know of works, namely our brain. So I basically had a career choice between studying the universe or studying the brain, which are probably the world's two greatest outstanding mysteries. I decided to take my chances on the brain.

At the time, the field of cognitive neuroscience was based on the paradigm that the brain is like a digital computer, and brain processes were modeled in essentially in the same way. There were lots of studies of memory, such as recalling lists of words, but very little was known about how the brain assigns a greater value to some things than others. But it's a really fundamental question, because the ability to work out whether something is good or bad—and to maximize behaviors that lead to good things and avoid bad things—is critical for survival. Digital computers typically don't make value judgments of that sort unless they are programmed to do so. So that's what excited me, trying to unlock how it is that the brain assigns value to things in the world.


Named for the late Caltech professor Earnest C. Watson, who founded the series in 1922, the Watson Lectures present Caltech and JPL researchers describing their work to the public. Many past Watson Lectures are available online at Caltech's iTunes U site.

Douglas Smith
Home Page Title: 
Learning From Experience — How Do We Do It?
Listing Title: 
Learning From Experience — How Do We Do It?
Exclude from News Hub: 
Short Title: 
Learning from the Past
News Type: 
Research News
Friday, February 13, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

Backpocket Barnburner: A Lightning Quick Overview of Educational Theory

Saturday, January 24, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

The personal side of science

Wednesday, February 4, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

Meet the Outreach Guys: James & Julius

Wednesday, February 18, 2015
Center for Student Services 360 (Workshop Space) – Center for Student Services

HALF TIME: A Mid-Quarter Meetup for TAs

Modeling Politics: An Interview With Alexander Hirsch

It is easy to assert offhandedly that the president made a particular decision in order to undermine Congress, or that Congress's latest bill is simply an attempt to bypass the Supreme Court. It is much harder to prove whether such arguments are accurate. After all, it is usually impossible to test such claims in a controlled manner. Still, some political scientists—such as Alexander Hirsch, a new associate professor of political science at Caltech—are using tools first developed by economists to do just that, creating mathematical models of these arguments. The models then serve as virtual laboratories that political scientists can use to test assumptions and identify implications.

Before coming to Caltech in August, Hirsch was an assistant professor of politics and public affairs at Princeton University. He earned his undergraduate degree in political science and economics at Yale University in 2003 and completed his doctoral work in political economics at Stanford Graduate School of Business in 2010.

Hirsch can often be found on campus with his dog, a goldendoodle named Baci, nearby. We recently sat down to talk with him about using models to study political behavior, why he is thrilled to be at Caltech, and where his research is headed.


What is the focus of your work?

I study simple, or toy, mathematical models of strategic behavior—game theoretic models as developed in economics, applied to political questions. The way I think about this kind of work is that when people study politics or they study elections, or they say that Obama chose to do this or the Republican Party is doing that, they're just making arguments about causal relationships. And those arguments are not really any different from the arguments that scientists make—you have assumptions, and you have implications. A and b implies c. If Obama wants a and thinks the Republican Party operates in b way, it suggests that c is the right strategic action for him to take. Those of us who use models are just subjecting those kinds of arguments to some mathematical rigor.


What do you gain from such modeling?

I think this is a valuable approach for a few reasons: First, people can be very sloppy about the arguments they make about why politics operates the way it does. Formalizing your logic forces you to be much more careful.

Writing down models also helps you discover the implications of your assumptions. People make assumptions about the preferences of political actors or how political institutions operate, and those assumptions have implications. The value of the model is not only to check whether those implications are valid, it is also to derive new implications that you didn't even imagine until you wrote down the model. Then you can use the data to actually see whether those implications come to pass.


What brought you to Caltech?

There are several things. One is that although many people don't know it, Caltech incubated this type of political-science modeling work early on, and the intellectual giants of the field were—or, in several cases are—here. It was an honor to have been invited to come and participate in that work.

One of the most appealing things to me about Caltech is that unlike many institutions of higher learning, Caltech lives and breathes research. There are other institutions that are wonderful at teaching and wonderful at research. But at Caltech it seems to me that to be wonderful at teaching means that you teach the students what the frontier of research is—that teaching and research are not different things.

An anecdote from teaching my first quarter here illustrates my point. It was around midterm time, and the students came in looking really exhausted. So I said, "Oh, I guess it's midterm time, so things are tough." And they said, "Well, actually, NSF grant deadlines are this week, so we were up all night."


Where does your passion for research come from?

My dad is a physicist and a theorist, so I grew up in a household in which academia was certainly respected as a profession. But the truth is that I wasn't really interested in pursuing an academic career for most of my childhood or even college. I gravitated a little bit more toward music and literature and stuff like that, none of which I pursued in a very serious way.

In college, I took some computer science courses and some math courses, but I was always interested in politics. Eventually, I started to take classes in economics, and I started seeing game theory taught by people who were interested in political applications. I found it very intellectually appealing that you could have these little mathematical toy models that look silly, but can say very surprising and in some cases deep things in this very controlled environment of the model.

So I saw economics and thought, "These tools are really cool and very rigorous and fun," and I saw political science and said, "These are interesting questions that I care about." So I started to gravitate in a direction where I could combine the two.


What are some of the specific topics you have researched using this approach?

I currently have a big research agenda on the incentives that legislators have to invest in developing expertise or developing new policy proposals. When will they want to put in a lot of effort versus very little effort to become an expert in an area or to develop new policies?

In one of my early papers, I also explored the idea that politicians, rather than disagreeing about what the aims of government should be, disagree about which policies will achieve those aims. I wrote down a model that said, let's imagine that politicians actually agree about what they want, but they disagree about how the world works—how to achieve those outcomes. So, for example, maybe Republicans and Democrats disagree about tax rates because they disagree about whether high taxes actually have a disincentive effect on work.


How do you write down a mathematical model about something like that?

Well, you're not trying to write a model that explains the world. You're just trying to write down the simplest possible structure that captures the types of political behaviors that you think are interesting.

In this case, the model that I wrote included just two possible policies: a and b. In the real world, of course, there are lots of possible policies, but we don't need to model all of those to try to understand what some of the political incentives are. In the model, there are two outcomes that can result from each of those policies—success or failure. I assign a utility of zero if the policy fails, and some number greater than zero if it succeeds. Then you need people to have the opportunity to learn, so you model a game in which people make the decision twice. They make their first choice and see what happened, then they use Bayes's rule—a simple mathematical theorem—to update their beliefs about which of the two alternatives is the right one, and then make the decision again.

The point is to predict patterns of what might come out of these incentives. It's not like modeling an atmospheric system; I'm not going to be able to make very precise estimates about what Congress is going to do tomorrow. It's to try to understand how these pieces fit together and what these assumptions might imply about the behavior of political actors.


What questions are you currently modeling?

There is an extensive literature in economics and in political science that tries to understand the nature of lobbying. Why does lobbying work the way it does? Why is lobbying effective? What are interest groups doing that legislators are responding to?

But there isn't a lot of literature on the lobbyists themselves. I am starting a project with Pablo Montagnes of the University of Chicago to try to understand the strategic incentives of lobbyists. Part of the focus is to try to understand the importance of the political ideology of a lobbyist. If you look at the available data, lobbyists look like passionate participants in the political process who have very real preferences about what government should look like, and not just like mercenaries who are willing to take sacks of cash to represent any interest group.

It's still very early, and we're working on the model. But we think that ideology plays a big role in what preserves a lobbyist's ability to represent you while also getting cash for it—it's what protects them from the incentive to take cash from anybody.


What do you do outside of work?

We have a very domestic life. My wife, Melanie, and I hang out with the junior faculty here. We like to eat at good restaurants. We've also been hiking a lot since we moved to L.A. We've hiked Echo Mountain maybe 12 times already. And we hang out with the dog.

Kimm Fesenmaier
Exclude from News Hub: 
News Type: 
In Our Community

Cake or Carrots? Timing May Decide What You'll Nosh On

When you open the refrigerator for a late-night snack, are you more likely to grab a slice of chocolate cake or a bag of carrot sticks? Your ability to exercise self-control—i.e., to settle for the carrots—may depend upon just how quickly your brain factors healthfulness into a decision, according to a recent study by Caltech neuroeconomists.

"In typical food choices, individuals need to consider attributes like health and taste in their decisions," says graduate student Nicolette Sullivan, lead author of the study, which appears in the December 15 issue of the journal Psychological Science. "What we wanted to find out was at what point the taste of the foods starts to become integrated into the choice process, and at what point health is integrated."

Since taste is a concrete, innate attribute—after all, people know what foods they like and do not like—the researchers hypothesized that it becomes factored into the food decision-making process first. A food's affect on health, on the other hand, is a more abstract attribute—one that you often need to learn about or do research on. In fact, there are such widely varying opinions about the healthfulness of nutrients like fats, calories, and carbs that you may not even be able to find a definitive answer. Therefore, the researchers assumed, the healthiness of a food likely is not factored into a person's food choice until after taste is. And for those individuals who exercised less self-control, they hypothesized, health would factor into the choice even later.

To test these ideas, Sullivan—along with her colleagues in the laboratory of Antonio Rangel, Bing Professor of Neuroscience, Behavioral Biology, and Economics, including Rangel himself—developed a new experimental technique that allowed them to evaluate, on a scale of milliseconds, when taste and health information kick in during the process of making a decision. They did this by tracking the movement of a computer mouse as a person makes a choice.

In the experiment, 28 hungry subjects—Caltech student-volunteers who had been fasting for four hours—were asked to rate 160 foods individually on a scale from –2 to 2, based on that food's healthfulness, its tastiness, and how much the subject would like to eat that food after the experiment was over. The subjects were then presented with 280 random pairings of those same foods and were asked to use a computer mouse to click on—to choose—which food they preferred from each pairing.

The researchers then used statistical tools to analyze each subject's cursor movements and, therefore, the choice process. They looked at how fast taste began to drive the mouse's movement—and how soon health did. For example, one subject's cursor trajectory might be driven by the taste of the foods very early in the trial, but soon after might be driven by health also—resulting in the selection of the healthier item, like Brussels sprouts over pizza. However, another subject's cursor trajectory might be driven by taste all the way to the selection of pizza—with health information coming online too late in the choice process to influence the selection of the food.

Sullivan and her colleagues found that, on average, taste information began to influence the trajectory of the mouse cursor, and thus the choice process, almost 200 milliseconds earlier than health information. For 32 percent of subjects, health never influenced their food choice at all; they made every single choice based on taste, and their cursor was never driven by the healthfulness of the items.

"What Nikki has shown is that a big factor here is how quickly you can represent and take into account different types of information when you are making choices," says Rangel. "People are making these choices very quickly—in a couple of seconds—so very small differences, even just a hundred milliseconds, can make an enormous difference in whether or how much health considerations ultimately influences the decision."

The researchers then wanted to find out if some people have an advantage in exercising self-control simply because they can factor health information into their choice earlier. Sullivan and her colleagues first split the subjects into two groups: those who exercised high self-control by often choosing the healthy option, and those who made their choices based almost entirely on taste—the low-self-control group.

On average, the low-self-control group began to factor in health information 323 milliseconds later than the high-self-control group. This suggests, says Sullivan, that the more quickly someone begins to consider a food's health benefits, the more likely they are to exert self-control by ultimately choosing the healthier food.

In addition, Sullivan says, it seems that those who calculate health earlier in the process also weigh it more heavily in their decision-making process.

These findings, she notes, mean it might one day be useful to encourage people to wait a bit longer before making a food choice. "Since we know that taste appears before health, we know that it has an advantage in the ultimate decision. However, once health comes online, if you wait—allowing the health information to accumulate for longer—that might give health a chance to catch up and influence the choice," she says.

Rangel adds that this work could also one day change the way health information is presented. "For example, if you go to the supermarket, does it matter how big the calorie count information label is on the yogurt?" he asks. "More visible information may affect how quickly you compute health information. We don't know, but this study opens such possibilities."

Sullivan and Rangel are next hoping to apply their cursor-tracking method to experiments beyond the refrigerator. They want to look, for instance, at how timing might affect self-control in choices involving saving money versus spending money, or deciding between an act of altruism versus an act of selfishness. They also plan to further explore the food-choice study in a larger and more diverse population of subjects through the Caltech Conte Center.

"In the past when psychologists and economists have thought about behavioral differences, they have thought of them as differences in preferences, like, 'Oh you make less healthy choices than her because you just value health less and that's the end of the story.' What our study is trying to say is that maybe part of these differences arise not from preferences, but from the amount of time it takes different people to represent information and feed it to the brain's decision-making circuit."

The Psychological Science study, "Dietary Self-Control Is Related to the Speed with Which Health and Taste Attributes Are Processed," was authored by Sullivan and Rangel along with Caltech postdoctoral scholar Cendri Hutcherson and former Caltech postdoctoral scholar and visiting associate Alison Harris, who is now an assistant professor of psychology at Claremont McKenna College. Their work was funded by the National Science Foundation.

Exclude from News Hub: 
News Type: 
Research News