A New Paradigm for Search: What Does It Cost to Run ChatGPT?

AI technologies such as Bard, which Google recently announced, will certainly make it look as if Google is dancing. Google clearly has some of the best AI research teams in the world, but until now it has not had a culture well suited to commercializing and productizing that AI capability. On top of that, the cost of operating AI-powered services is also a major issue.

A type of AI called an LLM (Large Language Model) became known to more than 100 million people with the advent of ChatGPT. LLMs deliver enormous value to people, but the future is not purely rosy. The biggest issue here is cost.

The cost of training LLMs has grown exponentially. That is because modern AI has advanced by scaling up complex MLP (Multi-layer Perceptron) models by tens of times every year.

The new Microsoft Bing, which applies a ChatGPT-style conversational client to search, looks highly innovative, but innovation never comes for free. As mentioned earlier, training an LLM is enormously expensive. Yet using an LLM actually costs far more than training it.

If an LLM like ChatGPT were used in search, it is estimated that about 20% of Google’s search revenue would be spent solely on operating the AI model.

Given that Google’s 2022 search revenue was $162.5 billion—more than 200 trillion won—it means that the cost of maintaining a conversational search engine would approach 40 trillion won a year.

According to semiconductor-focused research firm SemiAnalysis, Google’s current estimated revenue per query (search) is $0.0161 and its cost is $0.0106. That translates into an operating margin of 34%. For an IT company, that is an exceptionally high level.

However, if a conversational agent powered by an LLM like ChatGPT were integrated into the search engine, the additional cost per query would rise by $0.0036.

The costs analyzed by SemiAnalysis only add in the hardware expense (computing cost) of running the LLM, so the real figure could be even higher. The most important factor—labor costs—is not included.

Even under conservative assumptions, this would generate an additional $36 billion in annual costs, and the operating margin (OPM) of Google’s search business would be slashed from the current 34% to 12%, a one-third reduction.

Microsoft’s CEO says that from now on, the gross margin of the search business will fall forever.

Isn’t that fascinating? The CEO of a company that wants to launch a new product and compete anew in the search market is the one publicly declaring that he is going to blow up the market’s margins.

Even so, there is a reason Microsoft is approaching the search market this aggressively. Let’s remind ourselves once more of Google’s massive search revenue—about 203 trillion won a year.

If Microsoft could peel away just 1 percentage point at a time from Google’s 90% share of the search market, it could expect an additional 2 trillion won in annual revenue. The logic is that this is a market with ample profit opportunities even if margins fall from where they are now.

Nevertheless, it is hard to expect that Microsoft Bing, simply by loading a ChatGPT model, will secure a meaningful share of the search market. The reason lies in cost competitiveness.

The ChatGPT-style LLM that Microsoft is using is very expensive to operate. That is why the revamped Bing is still being offered only to a limited number of users.

As a giant that already dominates the search market, Google is rolling out a smaller, more cost-efficient LLM model to general users. There is still no clear evidence that a ChatGPT-style conversational agent has enough influence to meaningfully disrupt its dominance in search, and aside from a subscription model there is effectively no proven monetization model yet.

The immediate challenge Microsoft faces is the rising cost that comes with scaling. Because Bing is not as popular a search engine as Google, it attracts fewer advertisers. The pool of ad dollars it can tap is smaller than Google’s, so any increase in costs hits the profitability of its search business more directly. In other words, to grow market share, Microsoft is in a position where it has to effectively sell at a loss.

Google has secured default status as the search engine on Apple’s iOS Safari browser through an annual contract reportedly worth 20 billion dollars, and it has become the de facto “default” for many users through Chrome, the world’s most widely used browser, whose default search engine is also Google.

Microsoft has long pushed distribution of its Edge browser in an effort to chip away at Google’s search share with the Bing search service, and this time it is focusing on improving the search experience itself by bringing in ChatGPT’s technology. But for most people, a conversational search engine may still be something they simply do not need.