Three Lessons From Chatting About Strategy With ChatGPT

When Geoffrey Hinton, a pioneer of deep learning, quit Google recently, he made it clear that he is worried about the risks of artificial intelligence. He is not alone. Following the launch of ChatGPT-4, thousands of artificial intelligence experts signed a letter calling for a pause in the development of more-powerful AI systems.

On the other hand, excitement about the opportunities arising from large language models (LLMs) and their speed of adoption is unprecedented. Microsoft was quick to integrate the new technology into its Bing search engine, and the company’s founder, Bill Gates, stated that ChatGPT will “change our world” without necessarily putting jobs at risk.

While speculating about the future of AI is irresistible, the more practical question is how we can use it right now. Conversations about this are taking place in classrooms, newsrooms, and workplaces around the world.

As business strategists, we wanted to see what generative AI could add to our work. We explored this question through a series of experiments on different aspects of the strategy creation process. In each of the experiments, we put a realistic question of strategy to ChatGPT, followed by a lengthy back-and-forth to refine the initial responses. The intention was to understand how the tool can support ideation, experimentation, evaluation, and the building of stories — and where it falls down.

Three lessons emerged from these experiments.

1. Expect interesting input, not infallible recommendations.

In one of our experiments, we asked ChatGPT to suggest some disruptive business ideas for a large European transport provider. The chatbot suggested a personalized planning app, a ride-sharing service, hyperloop transportation, and a smart-luggage delivery service. Coincidently, the first three matched ideas from a recent workshop with a transport provider in another European country. The tool was also able to provide business models and cost estimates for these business ideas.

On the one hand, this is impressive. At the same time, it highlights that the tool seems unlikely to come up with ideas humans can’t, although it gets results faster and with less effort. Several other experiments confirmed this.

In another experiment, it also became obvious that humans are better at translating ideas into actions. For example, when we asked ChatGPT to come up with an idea for a new streaming service, the list of suggestions included a service focused on education. When we asked what it would take to realize such an idea, it proposed a partnership with either a university or an established consulting firm. While that made sense, ChatGPT’s suggestions for how to win over such partners were not sufficiently concrete and realistic. The current iteration of the tool, which is based on training set data in the public domain, isn’t able to capture the nuances of each corporate reality.

Still, LLMs can be helpful in the strategy process for a few key reasons:

They are fast and easy to use. You can avoid all the logistical effort required to get the right people in the room at the same time.

The tendency toward conventional thinking can be mitigated by asking the right follow-up questions. For example, when we asked for an innovative new concept for a bakery, ChatGPT initially suggested just offering savory products. We then asked for more creative ideas and got a space-themed bakery, with robotic assistants and bakers, zero-gravity dining, and 3D food printers, which was too expensive to realize. In a third attempt, the ideas became more interesting — for example, a bakery that uses artificial intelligence to analyze data about customer preferences and food trends to create and design unique flavor combinations for baked goods. From there, we were able to probe about steps to get started on realizing this idea.

It’s easy to generate many ideas. The most interesting ideas were, however, mixed with implausible, impractical, and untested ones. Only by sifting through them in a discerning manner can you get a better understanding of what might actually work.

2. Experienced strategists will benefit most from ChatGPT.

ChatGPT acts like a worldly wise and persuasive friend who is prone to overconfidence to the point of sometimes asserting fictions as facts, either because errors were present in the initial training set or through some emergent process that experts are still trying to understand.

While it would be foolish to rely unquestioningly on the advice of such a fallible friend, its suggestions could in fact be very useful in several ways, provided we fully apply our powers of judgment. We could tap into this friend’s extensive knowledge to characterize competitive environments. We could generate ideas upon which to base a strategy. We could simulate how different strategies might perform in different scenarios and highlight the risks and contingencies.

But extracting insights and avoiding being misled require that we exercise our powers of discretion by asking the right questions in the right sequence, judging the consistency of recommendations with our own knowledge and experience, checking the accuracy of assertions and assumptions, and judging the feasibility of the resulting strategies. In other words, we stand to benefit by applying the very same powers of discrimination that experienced strategists must regularly employ but novices are likely to lack.

The main attraction of ChatGPT in strategy work is probably its ability to write well, complementing the blind spot of many strategists. Strategists often underestimate the importance of how ideas are communicated, wrongly assuming that what they propose is what matters most. With only 28% of managers being able to correctly name three of their organization’s strategic priorities, this points to a misperception of what it takes to create impact in practice. But ChatGPT definitely has something to offer managers for whom creating engaging stories is both time-consuming and challenging. If you provide or refine the proposition, the tool can add style.

3. Strategists use different data and use it differently.

ChatGPT processes billions of data points to construct output. This might create the false impression that it exceeds the cognitive capacity of humans. It’s easy to overlook the limitations imposed by the lack of transparency into the exact nature of this data set and its gaps. Strategists often rely heavily on proprietary information, since effective strategies must uniquely reflect the position and capabilities of each company. We know for sure that current tools don’t have access to such inside knowledge. It is, therefore, not surprising that ChatGPT’s answers are somewhat generic — more or less according to the textbook, but lacking spice.

Most importantly, effective strategy often relies not on what is generally the case or generally effective but rather on anomalies to create new and unique approaches. Relatedly, as Giovanni Gavetti from the Tuck School of Business and Jan Rivkin from Harvard Business School explain, strategists often rely on analogies. This means they are able to make decisions with very little or no data that applies to a specific situation. ChatGPT doesn’t seem to have the capability to infer useful counterfactuals using analogical reasoning.

Even if LLMs were to become capable of asserting all of the facts, as opposed to linguistic correlations in a constrained data set, they would still face a limitation. Experienced strategists know to be skeptical of “facts.” What is true most of the time may not be true in a specific case. What was true in the past may not be true in the future. What happens to be true may not be inevitably and persistently so. And even if a statement is robustly true in these senses, there are always alternative ways of framing problems and solutions.

Strategists know that this framing step is often the most subtle and important one in developing a strategy. Strategy is, in one important sense, as much about useful fictions as established facts. By aiming to shape circumstances to bring about a new goal, strategies create new facts. In this regard, strategists are in the business of distinguishing useful fictions from misleading or infeasible ones — again a skill that only experienced strategists are likely to possess.

An interesting question is whether this pattern of utilities and limitations is likely to remain stable as the technology progresses. To assess this, we conducted the same experiments with ChatGPT-3 and ChatGPT-4. The substance of the answers remained quite similar. To some extent, the limitations were even more pronounced in the more advanced version of the tool, whose answers were, on average, twice as long. The newer version seemed to be a little more careful, like a slightly older friend who has been caught out before and wants to hedge their opinions. This manifests itself with a taste for pro and con lists and less pronounced judgment — a feature we look for in a strategist.

---

The bottom line: ChatGPT and similar tools are more likely to be useful in particular steps of strategizing (like idea generation and storytelling) and to experienced strategists rather than naive beginners. Such tools are, therefore, no substitute for the cultivation of strategic minds. And just as airline pilots must avoid over-relying on an autopilot, would-be strategists must avoid impeding the development of their own discriminating powers.