Is mamba-trwnsformer hybrid transformer like? Falcon H1 is an example of such architecture.

@ProjectVictory does it use some kind of attention mechanism?

What constitutes Transformer-like?

Is it all, or some, of these?

• Token inputs, token outputs

• Positional encoding

• (MH QKV SDPA -> MLP) layers

• Residual connections

https://arxiv.org/abs/2501.00663

Titans: Learning to Memorize at Test Time

Abstract page for arXiv paper 2501.00663: Titans: Learning to Memorize at Test Time

@JaundicedBaboon Let's see if it can leave the academia limbo where the Mamba's got stuck...

https://x.com/_philschmid/status/1846452029582959012

@MalachiteEagle Transformers, yes. Attention, not really:

followed by two shared attention blocks.

https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers

It even links the wager page in its bragging points:

All while being an “Attention-Free Transformer”

I thought this wager was made today haha

Does this strictly resolve to the outcome of the bet?

@FranekZak Yep. Whatever details they come up with are in.