This week's "Conversations With Bill Kristol" interview is with Jim Manzi, who talks about the value of experimentation in business and public policy:
[I] left and went to work for a little group that had spun out of the Boston Consulting Group, which is a strategy consulting firm. And this little spin-off firm was focused on the idea of using data and fairly intensive analytics, which at the time, this is 1987, which was a pretty innovative idea, to do really high-level strategy corporate work. And we very quickly started getting deep into using mathematical models to understand how to answer the kinds of questions I was referencing before, we're now answering experimentally. Would the following major investment in redesign of the bank branches actually pay out or not, would we create enough incremental customer volume to pay for this or not.
And I remember well standing in a conference room talking to a guy who was a partner when I was a very young consultant and laying out with great pride this kind of very complicated program of sophisticated conjoined analysis and cost analysis and all kinds of other things to answer this question for this large bank: would you actually get enough incremental business at the bank to justify this major rehab of the branches and change in employee behavior and so on? And he kind of politely listened and then at the end, said, "Okay, but why wouldn't you just try it at a few branches, you know and see if it works or not?" And I kept trying to, you know, respond to him, and I'd get halfway through the sentence in which I was explaining why that was naïve and not nearly as smart as what I was describing and then I'd kind of stop and say, "No, no, that's not true." Finally, I kind of gave up and said, "You know you're right." And that conversation needled at me for years and years and years....
It turns out the guy, the partner who told me you should try things was Rich Fairbanks and he and another guy who also worked there named Nigel Morris started a company called CapitalOne, the huge credit card company, really, literally as a platform for large-scale randomized experimentation. That really was the concept of what CapitalOne was, which of course is now a company worth tens of billions of dollars, so, you know, like he won the argument.