Any reason for the recent jump?

For context, how picky are you? How often do you buy a book and then decide it's not worth your time?

@AlexRosence5a I very rarely don't finish a book (or finish and think it was a waste of time). Maybe about once in every 20 or so? But I often base my reading on recommendations from people I trust, or books by authors I already like, so I'm not sure how useful that is as a metric. Normally if I read a terrible book, I already knew going into it that it was a long shot. And remember that for this market, I'll be giving it a prompt for the kind of book I'm after, so it's not going to get marked down just because I end up reading a book that's not my style.

I'm intending this market to be about "can it be done to a basic level of competence" rather than "does it surpass human authors".

@Fion That's useful information. Personally, I would not enjoy the average book that was done to a basic level of competence (by a human), and feel like it was not a good use of my time. So that phrasing was a bit misleading to me.

@AlexRosence5a Yeah, it's a difficult thing to convey without objective benchmarks. I now think I've given the wrong impression with my attempt at clarification. It definitely needs to be much better than what the average human could do, because the average human couldn't write a novel competently.

I once read a Brandon Sanderson YA novel. It made sense, it was quite exciting, but there were some aspects of the writing I didn't think were that great. (I've read other Brandon Sanderson novels and thought they were great, so I think he's a good writer, and his capacity for grand narratives is very impressive, but this particular book didn't blow me away.)

I haven't read Dan Brown, but he's a popular author who I'm pretty sure would pass the "basic competence" bar. Lots of people are very unimpressed by him, but he clearly knows how to write a book.

It's very difficult to give examples because quality is all so subjective. Maybe you actually think Dan Brown is great. Maybe you think he's horrendously awful and doesn't even qualify as "basically competent". But you must have read some "three star" books in your time? Were they a waste of time?

I get that there are lots of books out there, and they take a while to read, and there are probably enough "five star" books to last you a lifetime, so in that sense, maybe it's a waste of time to read anything that's only "good" or "very good". But I don't see things that way. I think something can be a good use of time even if it was only "good". (I feel like we're veering into general life philosophy now, though, so I'd better stop.)

But anyway, getting back to the market, I don't think I could reasonably make a market on "will AI be able to write books as good as my favourite books". I think that's much more subjective and difficult to judge than whether it will be able to write "good books".

@Fion Are you intimating here that Dan Brown would be approximately the low bar to clear?

@marvingardens I'm tempted to say "yes", but I don't think it's as simple as that. Somebody who loves Dan Brown and thinks he's one of the great authors of the generation would find my bar to be lower than Dan Brown. Somebody who thinks he's a terrible writer who can't string a narrative together or create consistent characters would find my bar to be higher than Dan Brown.

Based on my opinion (again, without having read anything) of Dan Brown as a fairly generic and unexciting, but broadly competent author... yeah, I think that's roughly where I'm putting my bar.

@Fion You could try this today. Ask ChatGPT for a short story in the form of the prompt in the market description. It'll write something coherent and competent (and likely uninteresting and wretchedly bland). How far from Dan Brown is that in your eyes? Personally, while I have no intention of ever reading a Dan Brown novel again, I would still say that the human thought that goes into his work makes it orders of magnitude better quality than what today's LLMs would produce. Hell, yesterday I read some truly godawful stories from a hundred-year-old pulp magazine, and I'd still say they easily and unimpeachably beat ChatGPT output just by having a point of view and an attempt at doing something, even when they straight up weren't coherent or competent. What's your own evaluation?

@marvingardens yeah, I agree that today's LLMs aren't close yet.

I'm reluctant to spend much time trying to evaluate short stories though. I don't really read short stories so I don't trust myself to be able to judge them by the same standards by which I'd judge a novel.

I pray not.

I think this is strictly easier than the movie market. So I don't see any reason not to sell some of my shares there to buy here when this one's trading lower.

@robm Great point. Though the movie version has more time than this one (by a year).

https://manifold.markets/ScottAlexander/in-2028-will-an-ai-be-able-to-gener

@dreev No it doesn’t. They only close two days apart. It’s part of scott alexander’s 5 year prediction post whose markets all resolve at the beginning of 2028

@DylanSlagh Ah, thanks! I jumped to the wrong conclusion from that "in 2028" in the title.

I doubt this will matter for resolution but I wonder if the AI could do much better by taking as many hours to write the thing as it takes you to read it, always staying one page ahead of you. From your perspective it's even more instant but the AI has hours or days or longer to keep making it better.

@dreev provided it was just one prompt and the user experience from then on was just reading, I'd count this. The important thing for this question is the user experience. The way in which that experience is achieved doesn't matter.

In outcomes where inference is slow/expensive for the highest-quality models, I can imagine it becoming commonplace for people to pay for inference runs that finish overnight for like $50. Compared to book prices nowadays, I don't know if this qualifies as "instant and cheap," but it's certainly fast enough and cheap enough that vast archives of these generations would be available. With respect to a highly personalized large product like a book, this would certainly be faster and cheaper than any similarly sophisticated custom product, and I think it actually meets the spirit of the question.

Overall, I'm worried about controversy in resolving this question in worlds where turbo models make mediocre novels ~instantly, but by waiting just a bit longer with the best models we get very compelling results.

@AdamK I would not resolve YES in the case of a $50 book overnight (in today's prices). But I would feel a bit uncomfortable. There is some amount of "spirit of the market" but I think that's too much. Then again, if we find ourselves in a situation where there are excellent books if you wait longer and pay more, and mediocre books that are cheap and instant, it depends on the quality of the mediocre books. I could see myself using the existence of the high quality ones as a kind of tie break.

Personally, my hope is that everything will be moving so fast that once we get to a $50 book overnight it's only a matter of months before we get a $1 book in ten minutes, so the chance of the end date falling in the awkward zone is small. But maybe that's naive.

quality is relative, if a system can infinitely produce passable mass market paperback tier writing, the marginal utility of that standard of writing diminishes, and people won't want to sit around and read it all day. it'll be similar to the feeling of generative content in video games that is all technically unique, but not intriguing enough to warrant continued interaction, even though relative to games of the past it is of much greater quality.

@brubsby that's a really good point! Maybe if I worry that's happened to me I'll need to reread some human-written books from today to recalibrate myself...

I'm a strong yes. I'd buy yes even if I knew that no huge LLM breakthroughs would happen in the next 4 years, just gradual incremental progress.

@jayharris Current context windows are at about few percents of a book in size. It seems realistic that they could increase by a couple orders of magnitude in four years.

@mariopasquato Hah 😅 more like 2 times a book in size and easyishly extendable

@TheBayesian It depends on the book I guess? Moby dick stands at 200k words, chatGPT has a context window of 8000 tokens (up to 32000) https://help.openai.com/en/articles/7127966-what-is-the-difference-between-the-gpt-4-models so we are at 4% (16%). Assuming 1 token per word, but it’s more than that typically.

@mariopasquato Gpt4 turbo has a 128k context window, and claude 2.1 has a 200k context window

@TheBayesian I see. Then in 4 years they will definitely be able to fit a full book in the window. Buying more YES