On Thursday, OpenAI released a $200-a-month chatbot — and the AI ​​community wasn’t quite sure what to make of it.

The company’s new ChatGPT Pro plan gives you access to “o1 pro mode,” which OpenAI says “uses more processing power to get the best answers to your toughest questions.” An improved version of OpenAI’s o1 reasoning model, o1 pro mode, should answer questions related to science, math and coding in a more “robust” and “comprehensive” way, OpenAI says.

Almost immediately, people began asking him to attract unicorns:

I asked ChatGPT mode o1 Pro to create a unicorn SVG.

(This is the model you’ve access to for $200 per month) pic.twitter.com/h9HwY3aYwU

— Rammy (@rammydev) December 5, 2024

And design a “crab-based” computer:

Finally introducing o1-pro for final use. pic.twitter.com/nX4JAjx71m

— Ethan Mollick (@emollick) December 6, 2024

And wax poetry concerning the meaning of life:

I just signed up for an OpenAI subscription for $200/month.
Please reply with inquiries to ask and I’ll re-post them on this thread. pic.twitter.com/oTQxbPxnoP

— Garrett Scott 🕳 (@thegarrettscott) December 5, 2024

But many people on X didn’t seem convinced that o1 pro replies were within the $200 range.

“Has OpenAI provided any specific examples of hints that failed in regular o1 but succeeded in o1-pro?” he asked British computer scientist Simon Willison. “I want to see a single specific example that shows his advantage.”

It’s a reasonable query; in spite of everything, it’s the most costly chatbot subscription on the planet. The service comes with other advantages, reminiscent of removal of rate limits and unlimited access to other OpenAI models. But $2,400 a 12 months is not bullshit, and the worth proposition of the o1 pro mode specifically stays unclear.

It didn’t take long to search out the failure cases. O1 pro mode has problems with Sudoku and is interrupted by an optical illusion joke that is clear to any human being.

Both o1 and o1-pro failed here, probably still as a consequence of vision limitations (same with Sudoku puzzles)https://t.co/mAVK7WxBrq pic.twitter.com/O9boSv7ZGt

— Tibor Blaho (@btibor91) December 5, 2024

OpenAI’s internal tests show that o1 pro mode performs only barely higher than standard o1 for coding and math problems:

OpenAI conducted a “more stringent” evaluation in the identical tests to reveal the consistency of o1 pro mode: a model was only considered to have solved a query if it answered accurately in 4 out of 4 cases. But even in these tests the development was not dramatic:

OpenAI CEO Sam Altman once wrote that OpenAI is on the correct track path “towards an intelligence too cheap to measure” – he was forced to achieve this explain many times on Thursday that ChatGPT Pro is not for most people.

“Most users will be very happy with o1 at (ChatGPT) Plus!” – he said in X. “Almost everyone will be best served by our free tier or Plus tier.”

So who is that this for? Are there really people who’re willing to pay $200 a month to ask questions on toys like “Write a 3-paragraph essay about strawberries without using the letter “e”” Or “solve this Mathematical Olympiad task“? Will they happily part with their hard-earned money, with little guarantee that the usual o1 won’t have the option to satisfactorily answer the identical questions?

I asked Ameet Talwalkar, associate professor of machine learning at Carnegie Mellon and enterprise partner at Amplify Partners, for his opinion. “I think it’s a big risk to raise the price tenfold,” he told TechCrunch by email. “I think in just a few weeks we’ll have a much better sense of the appetite for this functionality.”

UCLA computer scientist Guy Van den Broeck was more candid in his assessment. “I don’t know if this price makes sense,” he told TechCrunch, “and whether expensive reasoning models will be the norm.”

o1 is “better than most people at most tasks” because, yes, humans only exist in disembodied, amnestic, multi-spin chat interfaces https://t.co/zbLY2BG5pQ

— Aidan McLau (@aidan_mclau) December 6, 2024

The generous view is that that is a marketing mistake. Describing o1 pro mode as one of the best at solving “toughest problems” doesn’t say much to potential customers. Neither unclear statements about how the model can “think longer” and reveal “intelligence”. As Willison points out, without concrete examples of the supposedly improved capabilities, it’s hard to justify paying more in any respect, much less ten times the value.

is such a fun, really helpful prompt for an AI model that costs $2,400 per 12 months

I hope Openai keeps these template tooltips until asi pic.twitter.com/JQ5vLKxWWR

— Dean W. Ball (@deanwball) December 6, 2024

As far as I do know, the target market is experts in specialized fields. OpenAI says it plans to supply a handful of medical researchers from “leading institutions” with free access to ChatGPT Pro, which is able to include o1 pro mode. Errors are of paramount importance in healthcare and, as Bob McGrew, former research director of OpenAI, stated: excellent on X, greater reliability might be the important unlock of o1 pro mode.

I play a bit with o1 and o1-pro.

They are excellent and a little weird. They are also not intended for most people more often than not. You really must have particularly difficult problems to unravel to get value from it. But if you’ve such problems, it’s a very serious matter.

— Ethan Mollick (@emollick) December 5, 2024

McGrew too he thought o1 pro mode is an example of what he calls “intelligence overhang”: users (and maybe modelers) do not know learn how to extract value from “additional intelligence” as a consequence of the basic limitations of a easy, text-based interface. As with other OpenAI models, the one strategy to interact with o1 pro mode is thru ChatGPT, and – in response to McGrew – ChatGPT just isn’t perfect.

However, it’s also true that $200 sets high expectations. Judging by its early reception on social media, ChatGPT Pro is not exactly a hit.