MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
Plus
12
Ṁ11652027
12%
chance
1D
1W
1M
ALL
This question is managed and resolved by Manifold.
Get
1,000
and3.00
Sort by:
I think it will resolve as NO because:
People will lose interest in MMLU after around 95%, so it's possible that GPT-x or whatever SOTA there will be will not even be evaluated on it.
There are definitely at least some ambiguous or simply incorrect questions in MMLU, I am not sure how much but I will not be surprised if it's close to 1%.
Reaching 99% is usually much harder than 98%.
Metaculus predicts 93% in mid-2025, and I personally predict 91.5%. Will still be a long way to go.
Related questions
Related questions
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?
16% chance
Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?
71% chance
What will be true of the SOTA AI on the FrontierMath benchmark, before 2026?
What will be true of the SOTA AI on the FrontierMath benchmark, before 2027?
Will the transformer architecture be replaced in SOTA LLMs by 2028?
66% chance
By 2026, will it be standard practice to sandbox SOTA LLMs?
29% chance
What will be true of the SOTA AI on the FrontierMath benchmark, before 2028?
BIG-bench accuracy 75% #4: Will SOTA for a single model on BIG-bench pass 75% by the start of 2027?
86% chance
BIG-bench accuracy 75% #3: Will SOTA for a single model on BIG-bench pass 75% by the start of 2026?
86% chance