Steve Levitt Explains How Testing Ideas Can Make Firms Better

Published on February 10, 2010

Until he was 17 years old, all Steve Levitt ever wanted to be was a professional golfer. But when he got paired against a 13-year-old who drove the ball 40 yards farther, he gave up his dream, said Levitt, William B. Ogden Distinguished Service Professor of Economics and director of the Becker Center for Price Theory.

Five years ago, Levitt returned to his first love and discovered a gadget that measured the speed of his club head. He added 50 yards to his drives. “Once I had this mechanism, I experimented,” he told students during a Becker Brown Bag Series talk at Harper Center on February 10. “I would try crazy things that I thought couldn’t possibly work. All sorts of things that were the opposite of what felt right turned out to be the best. All of my golf instincts turned out to be off.”

Imagine trying to apply the same principles to a business firm filled with experts in their fields, Levitt said. “A golf swing is really not complicated compared to mapping what individuals do at firms and the outcomes,” he said. “There are so many things going on, including what competitors are doing, what the macro-economy is doing, advertising, and sales people. The beauty of an experiment is trying to cut through all that.”

In a complicated situation, randomization is an experimenter’s best friend, Levitt said. “If you randomize, you can become as complex as you want,” he said. “That is a simple idea that has underlined the scientific method for hundreds of years.”

What It Takes for Firms to Experiment

Unfortunately, most businesses don’t try such experiments, Levitt said. “One reason is that most people do in businesses what everybody else did before them,” he said. “You could say the experiments aren’t valuable and the market can figure these things out, but I don’t think that’s the answer. People have just not thought very hard about this.”

To develop experiments in firms, Levitt said, executives should take the following steps:

  • Decide what question needs to be answered—and why. “Use theory in some broad sense of the word or your experience, but figure out what you want to solve or go after and why,” he said.
  • Learn whatever is available from past experience. “Before you run an experiment, figure out what you already know,” Levitt said. “Many times, existing information can guide you.”
  • Design the experiment. “Think about what your treatment will be and what your control will be,” he said.
  • Execute the experiment well and analyze it correctly. “It should be really, really easy. And yet over and over, people analyze the wrong set of numbers,” Levitt said.

Levitt showed that experiments in firms are much less risky than launching full-scale projects, said second-year student Andrea Kenyon, a second-year student in the Full-Time MBA Program. “It’s more about trying to make firms better in general,” Kenyon said. “Whatever ways you can do that are important. Instead of just making decisions, testing ideas better serves the empirical principles of the Booth approach.”

Levitt said he wishes he had known that the 13-year-old he was paired against grew up to become PGA member Tim Herron. “He turned out to be one of the world’s greatest golfers, winning three tournaments in his first four years on the tour and making millions of dollars playing golf,” Levitt said. “If only I had more information then about who this guy was going to turn out to be, maybe I would have continued playing golf more aggressively.”

Phil Rockroh