The information leaders are receiving is still very useful to them in reaching decisions
Image: Shutterstock
From pilot projects to A/B tests, executives love to apply scientific methods to decisions. But companies don’t have years to spend on data analysis—they have to make clear goals and strategic choices now.
That’s why business experiments often turn into staged performances with scripted endings, says a recent study. Test designers might consciously or unconsciously advocate for an agenda instead of generating truly unbiased information. And that may not be such a bad thing.
Rebecca Karp, an assistant professor at Harvard Business School who focuses on how companies formulate and execute strategies, studied how experiments can play different roles in business. Not only can they be used to gather data and learn from it, but also to persuade audiences around key decisions.
“The information leaders are receiving is still very useful to them in reaching decisions,” says Karp, who coauthored the paper, “Business Experiments as Persuasion,” with Orie Shelef and Robert Wuebker, professors at the University of Utah’s Eccles School of Business. “The information can also be a useful tool for decision-makers when they have other audiences that they need to persuade.”
The research comes at a time of consumer skepticism. Nonetheless, the audience for such experiments tends to be other business leaders, and they don’t expect such experiments to be exhaustive. Business, after all, can’t boast the scientific rigor of other disciplines that lean on double-blind clinical trials or field research over multiple years.
“Decision-makers can and should view experiment results with a critical eye,” Karp explains.
What the ‘smark’ knows
In recent years, a number of studies have evaluated the merit of using scientific approaches to business decision-making. There’s a growing consensus that testing the validity of ideas can help a company decide how to reshape its business model, guide product innovation and investments, and steer training and hiring decisions. These include randomized controlled trials, iterative hypothesis testing, focus groups, and crowdfunded product development.
But some studies indicate that those designing and conducting experiments don’t always objectively set out to gather as much information as possible to help managers reach decisions. Instead, experiment designers often try to influence managers to reach certain conclusions to secure a strategic advantage—be that more resources or better access to a new technology.
Prior to her recent study, Karp says she personally observed the early stages of 13 pilot programs and experiments tied to the adoption of new technologies or other innovations at companies. In each case, she says, the business experiments aimed to produce results that reflected the views of the people overseeing them.
Karp leans on the language of professional wrestling to explain the phenomenon. For example, the “kayfabe”—a match that’s being staged—might be a presentation to a group of investors. A decision-maker is influencing a “smark”—or a smart mark—who is aware that the experiment is rigged as part of the staged performance. However, the smark still benefits from access to the information.
Also read: Understanding decision-making: Inherent risk preferences
When bias is ‘a feature, not a bug’
Karp devised a model to examine how a founder might use an experiment to convince a venture capitalist to invest. The founder could structure the experiment to favor a particular investment—a situation that the investor would expect in a pitch.
The study’s findings confirm what Karp had been observing in the field: Experimental processes shift from the objective to the subjective during the early stages of designing experiments.
Karp likened the bias inherent in business experiments to venture capitalists listening to funding pitches from entrepreneurs. The audience understands that persuasion is ultimately “a feature, not a bug” of the process of receiving information and reaching critical decisions, as one venture capitalist explained to her.
“The venture capitalists know that the experiment is likely designed to persuade,” says Karp. “The better you are at influencing—the venture capitalist knows that; they are looking for people who know how to play the game.”
Persuasion has some plusses
The researchers are not trying to dissuade executives from conducting experiments before making key business decisions, Karp says, stressing how critical it is for companies to carefully research and analyze options when confronted with challenges. Nor are researchers encouraging decision-makers to pass judgment on those whose experiments almost inevitably veer from objective analysis to subjective persuasion.
Instead, decision-makers need to accept that those designing and conducting experiments are human—and that no experiment is perfect nor free from some form of bias. Experiments designed to persuade their audiences are still worthwhile, Karp says, as long as decision-makers understand their limits.
Indeed, storytelling has become a powerful communication skill for leaders today. The right experiment might lead to a compelling narrative that helps a startup land its next investment or a company use the right technology vendor, the research indicates.
This article was provided with permission from Harvard Business School Working Knowledge.
