Think Healthy, Eat Healthy: Caltech Scientists Show Link Between Attention, Self-Control

PASADENA, Calif.—You're trying to decide what to eat for dinner. Should it be the chicken and broccoli? The super-sized fast-food burger? Skip it entirely and just get some Rocky Road?

Making that choice, it turns out, is a complex neurological exercise. But, according to researchers from the California Institute of Technology (Caltech), it's one that can be influenced by a simple shifting of attention toward the healthy side of life. And that shift may provide strategies to help us all make healthier choices—not just in terms of the foods we eat, but in other areas, like whether or not we pick up a cigarette.

Their research is described in a paper published in the July 27 issue of the Journal of Neuroscience.

When you decide what to eat, not only does your brain need to figure out how it feels about a food's taste versus its health benefits versus its size or even its packaging, but it needs to decide the importance of each of those attributes relative to the others. And it needs to do all of this more-or-less instantaneously.

Antonio Rangel, professor of economics and neuroscience at Caltech, has been studying this value-deriving and decision-making process for years now. Along with Todd Hare—a former postdoc at Caltech who is now an assistant professor of neuroeconomics at the University of Zurich in Switzerland—he published a paper in Science in 2009 describing differences in the brains of people who are better at exercising self-control than others. What they found was that while everyone uses the same area of the brain—the ventral medial prefrontal cortex, or vmPFC—to make value-laden decisions like what to munch on, there's a second brain area—the dorsolateral prefrontal cortex, or dlPFC—that seems to come to life when a person is using self-control during the decision-making process.

In other words, when the dlPFC is active, it allows the vmPFC to take into account health benefits as well as taste when it assigns a value to a particular food.

The new study goes a step further, showing that there seem to be ways to help kickstart the dlPFC through the use of what Hare calls "external cues" that allow us to exhibit more self-control than we might have otherwise.

The researchers came to their conclusions based on data from a brain-imaging experiment conducted with 33 adult volunteers, none of whom were following a specific diet or trying to lose weight for any reason. Each of the volunteers was shown 180 different food items—from chips and candy bars to apples and broccoli—through a set of video goggles while in a functional magnetic resonance imaging (fMRI) machine.

The hungry subjects—they were asked to fast for at least three hours prior to the experiment—were given up to three seconds to respond to each picture with a decision about whether or not they'd want to eat the food shown after the experiment was over. They could either give the food a "strong no," a "no," a "yes," or a "strong yes." Once all of the images had been flipped through, a single food image was chosen at random; if the volunteer had said "yes" or "strong yes" to the idea of eating that food, he or she was served that item.

"Because only one random trial was selected to 'count,'" says Rangel, "the optimal strategy for subjects is to treat each decision as if it were the only one."

Simple, right? But here's the catch: before every 10 food choices, an instruction would come on the screen for five seconds telling the subjects either to "consider the healthiness," "consider the tastiness," or "make decisions naturally." This meant that of the 180 decisions, the subjects made 60 in each of the three "instruction conditions."

What this was meant to do, Rangel explains, is shift the subject's attention during the experiment and, potentially, shift the way in which they made decisions.

Afterward—outside the scanner—the subjects were asked to rate the same foods on both a tastiness scale (very untasty, untasty, tasty, very tasty) and a healthiness scale (very unhealthy, unhealthy, healthy, very healthy). That way, the researchers were able to associate the choices the subjects made during the brain scan with their stated perceptions of those foods' attributes—showing that a subject who chose broccoli during the "consider the healthiness" portion of the test might think of it nonetheless as untasty.

The researchers then classified the foods for each subject based on that subject's ratings: unhealthy-untasty, healthy-untasty, unhealthy-tasty, and healthy-tasty. Unsurprisingly, people chose healthy-tasty foods no matter where their attention had been directed.

Things got interesting when the researchers looked at the other three categories, however. Among their findings:

  • When thinking about healthiness, subjects were less likely to eat unhealthy foods, whether or not they deemed them to be tasty, and more likely to eat healthy-untasty foods.
  • Being asked to think about healthiness led subjects to say "no" to foods more often than they did when asked to make decisions naturally.
  • There were no real differences between the choices made during the "consider the tastiness" and "make decisions naturally" portions of the experiment.

When the researchers turned to the fMRI results, they found that the vmPFC was, as predicted, "more responsive to the healthiness of food in the presence of health cues," says Rangel. And, as they'd seen previously, the robustness of that response was due to the influence of the dlPFC—that bastion of self-control—which was much quieter when the study's subjects were thinking about taste or their own personal choice than when they were asked to throw healthiness into the equation.

"This increased influence of the health signals on the vmPFC results in an overall value for the food that is based more on its health properties than is the case when the subject's attention is not focused on healthiness," says Hare.

These results are most likely not limited just to choices about food, Hare says. "Our findings are also relevant to the current changes to cigarette warnings many governments have started to make," he notes. "These changes include adding graphical images of the health risks of smoking. It remains to be seen whether these images will be more effective in drawing attention to the unhealthiness of smoking than the text warnings. If the graphical warnings do increase attention to health, then our results suggest that they could decrease the desire to smoke."

Jonathan Malmaud, a former research assistant at Caltech who is now a graduate student at MIT, was also an author on the Journal of Neuroscience paper, "Focusing attention on the health aspects of foods changes value signals in the vmPFC and improves dietary choice." The scientists' work was funded by a grant from the National Science Foundation.

Lori Oliwenstein

Caltech Researchers Pinpoint Brain Region That Influences Gambling Decisions

PASADENA, Calif.—When a group of gamblers gather around a roulette table, individual players are likely to have different reasons for betting on certain numbers. Some may play a "lucky" number that has given them positive results in the past—a strategy called reinforcement learning. Others may check out the recent history of winning colors or numbers to try and decipher a pattern. Betting on the belief that a certain outcome is "due" based on past events is called the gambler's fallacy.

Recently, researchers at the California Institute of Technology (Caltech) and Ireland's Trinity College Dublin hedged their bets—and came out winners—when they proposed that a certain region of the brain drives these different types of decision-making behaviors.

"Through our study, we found a difference in activity in a region of the brain called the dorsal striatum depending on whether people were choosing according to reinforcement learning or the gambler's fallacy," says John O'Doherty, professor of psychology at Caltech and adjunct professor of psychology at Trinity College Dublin. "This finding suggests that the dorsal striatum is particularly involved in driving reinforcement-learning behaviors."

In addition, the work, described in the April 27 issue of The Journal of Neuroscience, suggests that people who choose based on the gambler's fallacy may be doing so because at the time of the choice they are not taking into account what they had previously learned or observed.

The focus of O'Doherty's research is to understand the brain mechanisms that underlie the decisions people make in the real world. To study this kind of decision making in the lab, his team gets study participants to play simple games in which they make choices that result in winning or losing small amounts of money. To make these games interesting, the researchers often present simple "gambling" scenarios, such as playing slot machines or roulette.

"For this particular study, we were interested in what part of the brain might play a role in controlling these strategies that drive behavior," says O'Doherty, who conducted the study along with postdoctoral scholar Ryan Jessup.

The team asked 31 participants to complete four roulette-wheel tasks while lying in an MRI scanner. For each round, the volunteers were asked to choose a color on a tricolored spinning wheel. If the wheel stopped on their color, they won two euros. (The study was done at Trinity College Dublin.) For each round, participants were charged a half euro, regardless of the outcome. All the while, the researchers studied the brain activity of participants, with a focus on how they appeared to choose colors.

"The dorsal striatum was more active in people who, at the time of choice, chose in accordance with reinforcement-learning principles compared to when they chose according to the gambler's fallacy," says Jessup. "This suggests that the same region involved in learning is also used at the time of choice."  

The two types of strategies are actually contradictory because in reinforcement-learning behavior, one would be more likely to choose something if it has won a lot recently, and less likely to choose something if it has lost a lot recently. The opposite is true of the gambler's fallacy.

"The task was novel because making decisions based on either reinforcement learning or the gambler's fallacy is not rational in this particular task, and yet most of the subjects acted irrationally," explains Jessup. "Only 8 out of 31 subjects were generally rational, meaning they simply chose the color that covered the largest area in that round."

"It is very important to try to understand how interactions between different brain areas result in different types of decision-making behavior," says O'Doherty. "Once we understand the basic mechanisms in healthy people, we can start to look at how these systems go wrong in patients who suffer from different diseases, such as psychiatric disorders or addiction, that impact their decision-making capabilities."

The study, "Human Dorsal Striatal Activity during Choice Discriminates Reinforcement Learning Behavior from the Gambler's Fallacy," was supported by a Science Foundation Ireland grant.

Katie Neith

Law Expert Wins Feynman Prize for Excellence in Teaching

J. Morgan Kousser, professor of history and social science at the California Institute of Technology (Caltech), has been awarded the Richard P. Feynman Prize for Excellence in Teaching—Caltech's most prestigious teaching honor.

Kousser was selected for his "exceptional ability to draw science and engineering students to appreciate the intellectual rigors of legal thought."

The Feynman Prize was established in 1993 "to honor annually a professor who demonstrates, in the broadest sense, unusual ability, creativity, and innovation in undergraduate and graduate classroom or laboratory teaching." Any member of the Caltech community, including faculty, students, postdoctoral scholars, alumni, and staff, may nominate a faculty member for the award, and the winner is selected by a committee appointed by the provost.

"Although people outside Caltech are sometimes shocked to find that we teach history and political science, English, economics, and philosophy, undergraduates here can get close attention from internationally known professors much more easily than at almost any other college in the U.S.," says Kousser. "Winning the Feynman Prize is a recognition of how much great teaching goes on in the humanities and social sciences division at Caltech and how central our division is to the undergraduate experience at Caltech."

A member of the Caltech faculty since 1969, Kousser is the author of The Shaping of Southern Politics: Suffrage Restriction and the Establishment of the One-Party South, 1880–1910 and Colorblind Injustice: Minority Voting Rights and the Undoing of the Second Reconstruction. His research focuses on minority voting rights, the history of education, and the legal and political aspects of race relations in the 19th and 20th centuries.

Kousser has served as an expert witness in 28 federal or state voting-rights cases and as a consultant in 10 others, and he testified before a subcommittee of the U.S. House of Representatives in 1981 about the renewal of the Voting Rights Act. He received his AB in 1965 from Princeton, and his MPhil and PhD from Yale University in 1968 and 1971, respectively. He also holds an honorary MA from Oxford University.

In nomination letters written by students, Kousser was commended for holding his students to high standards and driving them to excel as critical thinkers. Several students described him as one of the most inspiring and demanding instructors at the Institute. The award citation remarked that his passion for his subject matter has even "drawn students to change their career path to pursue law, a remarkable achievement in an environment so dominated by science and engineering."

"Under his tutelage, many Caltech students—myself included—grow from politics neophytes into judicial experts over the course of the two terms of Law 148," said Elizabeth Mak, a senior in biology, in her letter nominating Kousser for the prize. "Professor Kousser's unique teaching style hinges on the strength of the respect his students have for him. Simply put, he inspires his students."

Kousser says he enjoys teaching at Caltech because he has the opportunity to structure classes so that students cannot avoid teaching themselves. He also appreciates that Caltech students take up his challenges "with verve and brilliance."

"I get a prize every year—watching students grow not only in knowledge, but in fascination with topics they were barely aware of before," says Kousser. "The real prize is the light in their eyes."

Previous winners of the Feynman Prize have included Dennis Dougherty, George Grant Hoag Professor of Chemistry; Jehoshua (Shuki) Bruck, Gordon and Betty Moore Professor of Computation and Neural Systems and Electrical Engineering; and Zhen-Gang Wang, professor of chemical engineering.

Katie Neith
Exclude from News Hub: 

Caltech Receives $12.3 Million to Train Scientific Leaders in Business and Industry

The California Institute of Technology (Caltech) has announced the creation of the Ronald and Maxine Linde Institute of Economic and Management Sciences. The initiative will bring together the best scientific minds and the best quantitative business practices, permitting a distinctive and targeted educational opportunity for Caltech's students and providing cutting-edge research opportunities for Caltech's faculty.

The new institute will be funded with an $8.2 million endowment established by Ronald and Maxine Linde and a $4.1 million addition to the endowment from the Gordon and Betty Moore matching program.

"Caltech students benefit greatly from exposure to the intellectual tools to become responsible, capable entrepreneurs and business managers," says Caltech's president, Jean-Lou Chameau. "By establishing the Linde Institute of Economic and Management Sciences, Ronald and Maxine Linde will foster the future growth of education and research activities that will prepare our students to assume leadership roles in industry and academia."

The new institute will be multidisciplinary, building on the success of Caltech's current Business, Economics, and Management (BEM) program by organizing its education and research activities into a single entity. BEM is an undergraduate option launched in 2002 and administered by the Division of the Humanities and Social Sciences (HSS).

The Linde Institute will be led by Peter Bossaerts, the William D. Hacker Professor of Economics and Management and professor of finance at Caltech, and will bring together current faculty while enhancing the Institute's ability to recruit additional scholars working in these areas.

"The new institute will enable Caltech to build on the excellence of its current research in the interdisciplinary areas that impact economics and management," says Ronald Linde, vice chair of Caltech's Board of Trustees. "It also will allow Caltech to equip its students with the proper background and tools to excel should they choose to become entrepreneurs or should they become involved in technology management."

Caltech is one of the few institutions at which such a program could be attempted, Linde adds, "because it combines an exceptional student population with a faculty that has chosen to pursue only the most analytically rigorous approach to business, economics, and management education."

"Caltech has taken a unique approach to the study of business," says HSS division chair Jonathan N. Katz. "Differing from a traditional business school model that is based primarily on inductively studying cases, Caltech's BEM program is rigorous, quantitative, and highly interdisciplinary. It provides students with analytical and conceptual tools to succeed in a modern, volatile business environment. To our knowledge, Caltech is the only institution to apply this social-scientific approach to undergraduate business education."

However, he adds, the BEM program—which is currently one of the most popular majors for students at Caltech—has "reached a threshold," requiring additional support to create a stronger infrastructure and unite the faculty working both within BEM and across Caltech's scientific disciplines.

"The Linde Institute will create a forum for sharing resources and making connections across campus," says Katz. "And it will foster the future growth and development of education and research activities."

The Lindes have sponsored numerous other scientific and research-intensive activities at Caltech, as well as a research facilities challenge grant. In 2008, the Lindes established an $18 million endowment at Caltech to create the Ronald and Maxine Linde Center for Global Environmental Science, which will be housed in a renovated 1930s laboratory. The Linde + Robinson Laboratory is scheduled to open in early 2012.

Lori Oliwenstein
Exclude from News Hub: 
News Type: 
In Our Community

Caltech Neuroscientists Study "Fearless" Woman

Armed with tarantulas, snakes, and horror-movie clips, Caltech neuroscientists, together with collaborators at the University of Iowa and the University of Southern California (USC), have studied a woman who is unable to experience the emotion of fear.

The work, described in the December 16 online issue of the journal Current Biology, provides the first in-depth investigation of how the experience of fear depends on a specific brain region called the amygdala and offers new insight into our conscious experience of emotions. The amygdalae, which register rapid emotional reactions, are implicated in depression, anxiety, and autism.

The individual, a woman known as SM who has extensive damage to the amygdala on both sides of her brain—rendering the brain regions essentially non-functional—showed no fear when subjected to numerous situations that normally induce the emotion.

The research team measured fear behavior by videotaping the subject as she walked through a haunted house at Halloween, handled large tarantulas and snakes in an exotic pet store, and viewed frightening film clips from horror movies. Detailed questionnaires and interviews were used to explore her experience of fear in response to all these situations, and during the day-to-day events in her real life.

"The complete absence of any fear behavior, or any report of subjective fear experience, was truly remarkable," says Justin Feinstein, a graduate student in clinical psychology at the University of Iowa and first author on the study. "This was especially striking because SM can feel emotions other than fear normally. This was not a subtle impairment, but something that just hit you in the face."

The other members of the research team were Ralph Adolphs, the Bren Professor of Psychology and Neuroscience at Caltech; Antonio Damasio, Dornsife Professor of Neuroscience and Director of the Brain and Creativity Institute at USC; and Daniel Tranel, Professor of Psychology and Neurology at the University of Iowa. The investigators have a long history of collaboration using an expansive registry of brain lesion patients at the University of Iowa.

According to Adolphs, the findings are especially valuable in light of the large amount of research on the amygdala in animals showing that the brain region is important for fear behaviors and fear learning. "One thing you simply cannot measure definitively in animals is their conscious experience of fear," Adolphs argues. "This study really fills an important open question; we had all assumed that the amygdala would be important for many aspects of fear processing, but demonstrating it also plays a role in feeling fear was not trivial to do."

In addition, Tranel says, "these findings will be of special importance to understanding psychiatric illnesses in which the amygdala is thought to be dysfunctional, such as depression, PTSD, and phobias."

Kathy Svitil

Alvarez on Alaska

Alaska's controversial and still-undecided Senate race between Republican Joe Miller and current Republican Senator Lisa Murkowski—who Miller beat in the primary and who then mounted a write-in campaign—may come down to a court challenge over the 92,500 write-in ballots cast. In his latest election-blog entry, Michael Alvarez—co-director of the Caltech/MIT Voter Technology Project—notes that, despite the brouhaha, 89 percent of the write-in voters somehow "managed to get Murkowski's name spelled correctly." Alvarez says that no matter which way this particular contest eventually goes, the real lesson may be in the success of the Murkowski campaign's efforts "to inform Alaskan voters how to cast a correct write-in ballot," and how that information might be used in the future by other campaigns in other states.


Lori Oliwenstein

The Polls Are Closed . . .

Calling last Tuesday's elections "historic" and "one of the most important midterm elections in recent memory," Caltech political scientist R. Michael Alvarez weighed in on the contrasts between the national and local results in an op-ed entitled, "GOP Tidal Wave Stopped at State Line," published in Sunday's Pasadena Star-News.

What made the difference? A number of factors, says Alvarez. "[I]t is likely that California's electorate on Tuesday was distinct from the rest of the nation," he writes. "As I visited polling places on Tuesday, in many parts of Los Angeles County there were energized voters, and visible voter-mobilization efforts. Early analysis of exit poll data indicates that those efforts might have led to a significant increase in Latino participation, another possible reason Republicans failed throughout the state."

To read more of Alvarez's analysis of the election results, go to the Pasadena Star-News. Or, for more research, analysis, and commentary on the election, visit the Caltech/MIT Voting Technology Project blog.

Lori Oliwenstein
Exclude from News Hub: 
News Type: 
In Our Community

What You See Affects What You Want

Visual attention guides the choices we make.

When it comes to making choices, say Caltech's Antonio Rangel and his colleagues, much depends on which items catch—and keep—your eye.

"We're interested in how the brain makes simple choices, like which item to pick from a buffet table," says Rangel, professor of neuroscience and economics. "Why is it that when we look at the buffet table, our gaze shifts back and forth between the items in order to make a choice? What is the role of visual attention in all this?"

To find out, Rangel—along with Caltech postdoctoral scholar Ian Krajbich and Stanford University's Carrie Armel—designed a mathematical model to describe the impact of what they call "visual fixation" on the making of these sorts of choices. (Simply put, visual fixation is the amount of time you spend gazing in one direction or at one item versus another.) They recently published their research online in the journal Nature Neuroscience.

"The model makes very specific predictions about how the pattern of fixation is related to our choices," Rangel explains. "For example, after controlling for other variables, items that are looked at more should be chosen more often."

But are they? To make sure, the team carried out an eye-tracking experiment in which subjects were shown pairs of food items on a computer screen, allowed to look at the items for as long as they wanted, and then were asked to choose one or the other. While the subjects gazed, the researchers tracked the movements of their eyes.

Just as predicted, the item a subject looked at longest was the one he or she most often picked—nearly three-quarters of the time, in fact.

Interesting stuff on its own merit, right? It also has implications for the business world—putting increased emphasis on things like packaging and in-store displays. "[T]he model explains how cultural norms (for example, reading left to right) can interact with comparator processes to produce cultural choice biases," the scientists write. "These biases help to explain, for example, why shelf and computer screen space on the top-left is more valuable than other positions."

Lori Oliwenstein

Consumers Will Pay More for Goods They Can Touch, Caltech Researchers Say

PASADENA, Calif.—We've all heard the predictions: e-commerce is going to be the death of traditional commerce; online shopping spells the end of the neighborhood brick-and-mortar store.

While it's true that online commerce has had an impact on all types of retail stores, it's not time to bring out the wrecking ball quite yet, says a team of researchers from the California Institute of Technology (Caltech).

Their investigations into how subjects assign value to consumer goods—and how those values depend on the way in which those goods are presented—are being published in the September issue of the American Economic Review.

The question they address is at the heart of economics and marketing: Does the form in which an item is presented to consumers affect their willingness to pay for it?

Put more simply, says Antonio Rangel, professor of neuroscience and economics at Caltech, "At a restaurant, does it matter whether they simply list the name of the dessert, show a picture of the dessert, or bring the dessert cart around?"

Most behavioral theories assume that the form of the presentation should not matter, notes Caltech graduate student Benjamin Bushong. "Some models suggest that choices amongst objects shouldn't vary with their descriptions or by the procedure by which the choice is made," he says. "However, our experiments show that the form in which the items are presented matters a lot. In fact, our research measures in monetary terms just how much those different displays matter."

Initially, the Caltech team made these measurements by presenting foods to hungry subjects in three different forms: in a text-only format; in a high-resolution photograph; and in a tray placed in front of the subjects. "Then we measured their willingness to pay for the food," explains Rangel.

As it turned out, there was no difference between the values subjects put on the food depicted in the text and in the picture. But the bids on the food on the tray right in front of the subjects were an average of 50 percent higher than the bids on either of the other two presentations.

"We were quite surprised to find that the text display and the image display led to similar bids," admits Bushong. "Initially, we thought people would bid more in the face of more information or seemingly emotional content. This finding could explain why we don't see more pictorial menus in restaurants—they simply aren't worth the cost!"

While the food experiments' results were intriguing, says Rangel, "We couldn't stop there." After all, the smell of the food might have made it more appealing to the experiment's subjects. And so, to take that variable out of play, the team chose different "goods" to present—a variety of trinkets from the Caltech bookstore—and again measured the effect of display on willingness to pay.

The results were the same as during the food experiments. The subjects were willing to pay, on average, 50 percent more for items they could reach out and touch than for those presented in text or picture form. "We knew then that whatever is driving this effect is a more general response," says Rangel.

But what was driving the effect? The team's initial hypothesis was that the behavior is driven by a classic Pavlovian response. "Behavioral neuroscience suggests that when I put something appetizing in front of you, your brain activates motor programs that lead to your making contact with that item and consuming it," Rangel explains. "We hypothesized that if there's no way for you to touch the item, then the Pavlovian motor response would be absent, and your drive to consume the item thus significantly lessened."

To test this hypothesis, the team put up a plexiglass barrier between the subject and the items up for bid. And, as predicted, once the possibility of physical contact with the item had been extinguished, the value the subjects gave to that item dropped to the same level as the text- and picture-based items.

"Even if you don't touch the item," says Rangel, "the fact that it is physically present seems to be enough. This Pavlovian response is more likely to be deployed when making contact with the stimulus is a possibility."

What does all this mean in the real world? At the very least, it suggests that your local bookstore—where you can reach out and ruffle a paperback's pages—may have more staying power than e-commerce experts might think.

In addition to Rangel and Bushong, the coauthors on American Economic Review paper, "Pavlovian Processes in Consumer Choice: The Physical Presence of a Good Increases Willingness-to-pay," are former Caltech undergraduate student Lindsay King and Colin Camerer, the Robert Kirby Professor of Behavioral Economics. Their work was supported by the Gordon and Betty Moore Foundation.

Lori Oliwenstein

Learning Strategies are Associated with Distinct Neural Signatures

PASADENA, Calif.—The process of learning requires the sophisticated ability to constantly update our expectations of future rewards so we may make accurate predictions about those rewards in the face of a changing environment. Although exactly how the brain orchestrates this process remains unclear, a new study by researchers at the California Institute of Technology (Caltech) suggests that a combination of two distinct learning strategies guides our behavior. 

A paper about the work will appear in the May 27 issue of the journal Neuron.

One accepted learning strategy, called model-free learning, relies on trial-and-error comparisons between the reward we expect in a given situation and the reward we actually get. The result of this comparison is the generation of a "reward prediction error," which corresponds to that difference. For example, a reward prediction error might correspond to the difference between the projected monetary return on a financial investment and our real earnings.

In the second mechanism, called model-based learning, the brain generates a cognitive map of the environment that describes the relationship between different situations. "Model-based learning is associated with the generation of a 'state prediction error,' which represents the brain's level of surprise in a new situation given its current estimate of the environment," says Jan Gläscher, a postdoctoral scholar at Caltech and the lead author of the study.

"Think about a situation in which you always take the same route when driving home after work, but on a particular day the usual way is blocked due to construction work," Gläscher says. "A model-free learning system would be helplessly lost; it is only concerned with taking actions that in the past were rewarding, so if those actions are no longer available it wouldn't be able to decide where to go next. But a model-based system would be able to query its cognitive map and figure out an efficient detour using an alternative route."

"Although the simpler model-free learning mechanism has been well studied and its basic learning mechanism—which is driven by reward prediction errors—is relatively well understood, the mechanisms underlying the more sophisticated model-based learning system, with its rich adaptability and flexibility, are less well understood" says John P. O'Doherty, professor of psychology at Caltech and the Thomas N. Mitchell Professor of Cognitive Neuroscience at Trinity College in Dublin, Ireland.

To further characterize the neurological underpinnings of these two learning systems, Gläscher, O'Doherty, and their colleagues designed a computer-based decision-making task that allowed them to measure when and where the brain computes both reward and state prediction error signals, and to determine if the two types of errors actually produce different neural signatures. 

In the task, subjects had to make choices between a left and right movement that allowed them to shift between different "states"—denoted by graphical icons—in a virtual environment; the process is similar to that of navigating around in a simple video game. Each left-or-right choice made in this virtual environment led the subject to a new state. Their objective was to reach a particular goal state to obtain a monetary reward, "and their chances of ending up in that goal state strongly depended on the particular pattern of sequential choices they made," O'Doherty explains.

A model-based system can learn about the structure of the virtual environment and then use this information to compute the actions needed to get to the reward state, in a manner analogous to how a chess player might try to think through the sequential chess moves needed to win a match. A model-free system, on the other hand, would only learn to blindly choose those actions that gave reward in the past, without evaluating the consequences in the current situation.

Eighteen participants were scanned using functional magnetic resonance imaging as they learned the task. The brain scans showed the distinctive, previously characterized neural signature of reward prediction error—generated during model-free learning—in an area in the middle of the brain called the ventral striatum. During model-based learning, however, the neural signature of a state prediction error appeared in two different areas on the surface of the brain in the cerebral cortex: the intraparietal sulcus and the lateral prefrontal cortex. 

These observations suggest that two unique types of error signals are computed in the human brain, occur in different brain regions, and may represent separate computational strategies for guiding behavior. "A model-free system operates very effectively in situations that are highly automated and repetitive—for example, if I regularly take the same route home from work," Gläscher says, "whereas a model-based system, although requiring much greater brain-processing power, is able to adapt flexibly to novel situations, such as needing to find a new route following a roadblock." 

These two distinct learning mechanisms serve complementary roles in controlling human behavior, Gläscher says. "Because the processing power of our brains is limited, it doesn't make sense to deploy the more computationally intensive model-based system for controlling everything we do. Instead, it is better to rely on the model-free system for a lot of our everyday behavior and use the model-based system only for new or complex situations. An important area for further research will be to try to understand the factors governing how these systems interact together in order to control behavior, and to determine how this is implemented in the brain."

The other coauthors on the paper, "States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning," are Nathaniel Daw of New York University and Peter Dayan of University College London. The work was supported by the Gordon and Betty Moore Foundation, the National Institute of Mental Health, the German Academy of Natural Sciences Leopoldina, and the Gatsby Charitable Foundation.

Kathy Svitil


Subscribe to RSS - HSS