ChatGPT helps and concerns, business advisors, study finds

When Karim Lakhani started testing how ChatGPT impacted the work of elite business consultants last spring, he thought they would be happy with the tool. In a preliminary study of twenty employees, the language bot had helped them complete two hours of tasks in twenty minutes.

“I assumed they would think, like me, ‘Great! I can do so much more!” said dr. Lakhani, a professor at Harvard Business School.

Instead, the advisors had feelings of unease. They appreciated doing better work in less time. But ChatGPT’s fast work threatened their sense of themselves as highly qualified workers, and some feared relying on it too much. “They were very concerned and felt like this would belittle them and be like empty calories for their brains,” said Dr. Lakhani.

After these preliminary tests, Dr. Lakhani and his colleagues conducted a larger, controlled experiment to measure how ChatGPT would impact more than 750 white-collar workers. That study, currently under review in a scientific journal, indicated strongly mixed results in the consultants’ work product. ChatGPT greatly improved the speed and quality of work on a brainstorming task, but it led many consultants astray when doing more analytical work.

The survey also identified employees’ varying feelings about the tool. One participant compared it to the fire that Prometheus stole from the gods to help mortals. Another told Dr. Lakhani’s colleague Fabrizio Dell’Acqua said ChatGPT felt like junk food: hard to resist, easy to consume but ultimately bad for the consumer.

In the near future, language bots such as OpenAI’s ChatGPT, Meta’s Llama and Google’s Gemini are expected to take on many white-collar tasks such as copywriting, preparing legal instructions and drafting letters of recommendation. The research is one of the first to show how technology can influence real office work – and office staff.

“It’s a well-designed study, especially in an emerging field like this,” said Maryam Alavi, a professor at the Georgia Institute of Technology’s Scheller College of Business, who was not involved in the experiments. Dr. Alavi, who has studied the impact of new digital technology on employees and organizations, also noted that the study “really highlights how much more we have to learn.”

Management consultants from the Boston Consulting Group, one of the largest management consulting firms in the world, were recruited for the study. The company had banned its consultants from using AI bots in their work.

“We wanted it to involve a large group of real employees working on real tasks,” says François Candelon, the company’s director who helped design the experiments.

The volunteers were divided into two groups, each working on a different management consulting problem. Within each group, some consultants used ChatGPT after 30 minutes of training, some used it without instructions, and some did not use it.

One of the tasks was to brainstorm about a new type of shoe, sketch a convincing business plan for its production and write convincingly about it. Some researchers believed that only humans could perform such creative tasks.

They were wrong. The consultants using ChatGPT produced work that independent reviewers rated about 40 percent better on average. People who simply cut and pasted ChatGPT results were rated even higher than colleagues who combined its work with their own thoughts. And the AI-enabled consultants were more than 20 percent faster.

Studies this year from ChatGPT in legal analysis And white collar writing jobs have found that the bot helps low performers more than the most experienced people. Dr. Lakhani and his colleagues found the same effect in their study.

However, on a task that required reasoning based on evidence, ChatGPT was not helpful at all. In this group, volunteers were asked to provide advice to a company set up for the study. They had to interpret data from spreadsheets and relate it to mock transcripts of interviews with executives.

This is where ChatGPT has lulled employees into relying on it too much. People without help got the right answer 85 percent of the time. People who used ChatGPT without training scored just over 70 percent. Those who were trained did even worse, getting the answer only 60 percent of the time.

In interviews conducted after the experiment, “people told us they hadn’t checked it because it was so polished and looked so good,” says Hila Lifshitz-Assaf, a management professor at Warwick Business School in Britain.

Many consultants said ChatGPT made them uneasy about how the tool would change their profession and even their sense of themselves. Nearly three out of four participants told researchers they worried that using ChatGPT would cause their own creative muscles to atrophy, said Mr. Candelon of the Boston Consulting Group.

“If you haven’t had an existential crisis about this tool, then you haven’t used it much,” says another co-author, Ethan Mollick, a management professor at the Wharton School of the University of Pennsylvania.