Will a LLM trained with FP4 have competitive performance in 2 years time? | Manifold

Will a LLM trained with FP4 have competitive performance in 2 years time?

Plus

15

Ṁ1421

Jan 21

25%

chance

1D

1W

1M

ALL

"Currently, the technology for 4-bit training does not exists, but research looks promising and I expect the first high performance FP4 Large Language Model (LLM) with competitive predictive performance to be trained in 1-2 years time." (see: https://timdettmers.com/2023/01/16/which-gpu-for-deep-learning/)

Granted, the model must be open source for us to know, so the market will resolve based on publicly available information.

This question is managed and resolved by Manifold.

Get

1,000

and

3.00

Sort by:

predictedNO

Exclusively in FP4? Or does partially in FP4 count. What if the model is on average 60% FP4 over the course of training?

I guess you covered this with "trained in 4-bit (to some extent)"

predictedNO

https://arxiv.org/pdf/2212.09720.pdf

predictedNO

@NoaNabeshima This is ab post-training precision adjustments

Competitive with what? SOTA with fp16?

predictedNO

This seems important @typedfemale
Will this resolve YES if scaling laws suggest a 4-bit model would be competitive if compute-matched to a SOTA 16-bit model?

predictedNO

(but there isn't a trained SOTA 4-bit model)

@NoaNabeshima Yes, you need to be better than everything else, but be trained in 4-bit (to some extent)

@typedfemale Finetuned w 4 bit would trigger Yes? 80% of parameters in 4 bit would trigger Yes?

Related questions

Will an LLM break 1400 ELO on LMSys before February?

Will a publicly-available LLM achieve gold on IMO before 2026?

LLM Hallucination: Will an LLM score >90% on SimpleQA before 2026?

Will an LLM be able to solve the Self-Referential Aptitude Test before 2027?

Will LLM training costs fall 300x by 2028?

Will an LLM improve its own ability along some important metric well beyond the best trained LLMs before 2026?

Will LLMs mostly overcome the Reversal Curse by the end of 2025?

6 months from now will I judge that LLMs had already peaked by Nov 2024?

Will Apple release its own LLM on par with state of the art LLMs before 2026?

Will one of the major LLMs be capable of continual lifelong learning (learning from inference runs) by EOY 2025?

Related questions

Will an LLM break 1400 ELO on LMSys before February?

Will an LLM improve its own ability along some important metric well beyond the best trained LLMs before 2026?

Will a publicly-available LLM achieve gold on IMO before 2026?

Will LLMs mostly overcome the Reversal Curse by the end of 2025?

LLM Hallucination: Will an LLM score >90% on SimpleQA before 2026?

6 months from now will I judge that LLMs had already peaked by Nov 2024?

Will an LLM be able to solve the Self-Referential Aptitude Test before 2027?

Will Apple release its own LLM on par with state of the art LLMs before 2026?

Will LLM training costs fall 300x by 2028?

Will one of the major LLMs be capable of continual lifelong learning (learning from inference runs) by EOY 2025?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules