[This is Casey's medium-confidence prediction from the 12/22/23 episode of Hard Fork]
This market will resolve to yes if Google's Gemini Ultra or other state-of-the-art LLM is roughly equivalent to OpenAI's best publicly available LLM on December 31, 2024, and if Google's AI products have cut into ChatGPT's share of the consumer LLM chatbot market. Otherwise, it will resolve to no.
1,000
3.00Resolved this to no because only 1/2 of the resolution criteria were fulfilled. (Gemini-Exp-1206 is at the top of Chatbot Arena's leaderboard, but Google has not cut into ChatGPT's consumer LLM market share appreciably -- and, as we'll discuss on our upcoming predictions episode, shoving Gemini overviews into Google Search results pages doesn't count.)
@KevinRoose18ac I'm so sorry, I should've commented earlier. I wonder if it might be fair to argue that it has cut into ChatGPT's market share (and I don't mean by having Google Search tell people to put glue on their pizza). Google Gemini on the Play Store has "50M+" downloads, and ChatGPT on the Play Store has "100M+". Gemini still has less than ChatGPT, but by my reading it would be fair to say that it meets the criterion of having "cut into ChatGPT's share of the consumer LLM chatbot market".
If you feel convinced, do you think it might be worth unresolving and re-resolving to Yes?
@yetforever I mean these both seem like free apps, do we have any evidence that the sales on these apps have this ratio, or that sales through the apps are significant fraction of sales overall?
@BoltonBailey It's hard to say either way, because Google hasn't (recently) published paid subscriber counts. In Feb 2024 Google One (without AI) hit 100M subscribers, but I'm not sure how to tell how many of those have upgraded to the AI plan, or how many new AI subscribers they've gained. That's why I just went by the app install counts.
@KevinRoose18ac I strongly disagree that Google didn't cut into chatgpt market lead here, but it is what it is.
cut into ChatGPT's share of the consumer LLM chatbot market
@spiderduckpig this passage does not actually mean it needs to beat openai in market share
@Bayesian I think the
>neutralize ChatGPT's lead
In the title implies that Gemini must remove ChatGPT's lead in consumer share, implying Gemini equalizes or overtakes ChatGPT. You are right that the wording below is less clear, but I feel like "cutting into ChatGPT's market share is very vague." Would Gemini getting even 1% of market share count as cutting into ChatGPT's market share?
@JohnCarpenter In my opinion it seems kind of stretching the definition of consumer market to describe Gemini in Google search as cutting into ChatGPT's consumer market. "Market" implies goods and services being purchased and sold, and Google Search is not a product sold to consumers, it is free for Google users to use. In contrast, users paying for ChatGPT model access are directly purchasing a consumer LLM. You might say that Google search is a product sold to the businesses who use it to advertise, but there's another issue: Gemini in Google search is not a chatbot. It uses an LLM to provide additional information for Google searches but it is not an LLM chatbot in that you cannot talk back and forth with it. The description specifies:
>consumer LLM chatbot market
And I feel like the spirit of that is referring to chatbots like Microsoft copilot, ChatGPT, Gemini as a chatbot (not in Google search), etc.
implying Gemini equalizes or overtakes ChatGPT
no. the description's resolution criterion is what matters, not the title's
@spiderduckpig Gemini market share has dropped so far from 17 to 13 over the year. If it went to even 18, that would seem like it was cutting into openai, but it has literally fallen
@JohnCarpenter https://www.zdnet.com/article/the-most-popular-ai-tools-of-2024-and-what-that-even-means/
And
https://firstpagesage.com/reports/top-generative-ai-chatbots/
And
https://explodingtopics.com/blog/chatbot-statistics
All seem useful. Gen AI Search results, at least so far, are not interactive
@Bayesian Okay, if the description's resolution criterion matters, what would you say the definition of "cutting into" is, then? If my interpretation of "removing ChatGPT's lead" is incorrect, what would count as Gemini cutting into ChatGPT's lead?