Recommend a Movie, Win a Million Bucks

If you liked Pulp Fiction, you'll probably love Kill Bill. But if you hate Star Trek, will you like Sleepless in Seattle? The Netflix website offers over 100,000 movies to rent, which is an awful lot of titles to browse if you don't know what you're looking for. With no clerk at the checkout counter to size you up and steer you in the right direction, it's up to the site's software to help you make your choice. When you log in, the system analyzes the ratings you've given to movies in the past and tries to discern patterns that will help it predict what you might be in the mood for now.

Machine learning, as this is called, is a very hot field, and recommendation systems lie at the heart of e-commerce. Still, there's a lot of room for improvement. It's easy to predict that someone who liked an ultraviolent Tarantino movie will probably enjoy more of the same, but trying to draw broader conclusions can be a very tricky business—as shown by this actual recommendation from another major e-tailer: "As someone who has purchased or rated The Philadelphia Story, you might like to know that Furry Hamsters from Hell is now available. You can order yours for just $19.95 by following the link below." (We're still shaking our heads over that one.)

In October 2006, Netflix offered a million-dollar prize to whoever could come up with the best recommendation software, as long as it beat the company's existing system by at least 10 percent. Announced over the Internet, the Netflix competition was essentially open to anybody in the world with access to a computer. The result was a creative frenzy that lasted for nearly three years and involved more than 5,000 teams from around the world. One of those teams included a Caltech alumnus. This is his story.