OpenAI launches an API for ChatGPT, plus dedicated capacity for enterprise customers

To call ChatGPT, the free text-generating AI developed by San Francisco-based startup OpenAI, a hit is a massive understatement.

As of December, ChatGPT had an estimated over 100 million monthly active users. It’s attracted major media attention and spawned countless memes on social media. It’s been used to write hundreds of ebooks in Amazon’s Kindle store. And it’s credited with coauthoring at least one scientific paper.

But OpenAI, being a business — albeit a capped-profit one — had to monetize ChatGPT somehow, lest investors get antsy. It took a step toward this with the launch of a premium service, ChatGPT Plus, in February. And it made a bigger move today, introducing an API that’ll allow any business to build ChatGPT tech into their apps, websites, products and services.

An API was always the plan. That’s according to Greg Brockman, the president and chairman of OpenAI (and also one of the co-founders). He spoke with me yesterday afternoon via a video call ahead of the launch of the ChatGPT API.

“It takes us a while to get these APIs to a certain quality level,” Brockman said. “I think it’s kind of this, like, just being able to meet the demand and the scale.”

Brockman says the ChatGPT API is powered by the same AI model behind OpenAI’s wildly popular ChatGPT, dubbed “gpt-3.5-turbo.” GPT-3.5 is the most powerful text-generating model OpenAI offers today through its API suite; the “turbo” moniker refers to an optimized, more responsive version of GPT-3.5 that OpenAI’s been quietly testing for ChatGPT.

Priced at $0.002 per 1,000 tokens, or about 750 words, Brockman claims that the API can drive a range of experiences including “non-chat” applications. Snap, Quizlet, Instacart and Shopify are among the early adopters.

The initial motivation behind developing gpt-3.5-turbo might’ve been to cut down on ChatGPT’s gargantuan compute costs. OpenAI CEO Sam Altman once called ChatGPT’s expenses “eye-watering,” estimating them at a few cents per chat in compute costs. (With over a million users, that presumably adds up quickly.)

But Brockman says that gpt-3.5-turbo is improved in other ways.

“If you’re building an AI-powered tutor, you never want the tutor to just give an answer to the student. You want it to always explain it and help them learn — that’s an example of the kind of system you should be able to build [with the API],” Brockman said. “We think this is going to be something that will just, like, make the API much more usable and accessible.”

The ChatGPT API underpins My AI, Snap’s recently announced chatbot for Snapchat+ subscribers, and Quizlet’s new Q-Chat virtual tutor feature. Shopify used the ChatGPT API to build a personalized assistant for shopping recommendations, while Instacart leveraged it to create Ask Instacart, an upcoming toll that’ll allow Instacart customers to ask about food and get “shoppable” answers informed by product data from the company’s retail partners.

“Grocery shopping can require a big mental load, with a lot of factors at play, like budget, health and nutrition, personal tastes, seasonality, culinary skills, prep time, and recipe inspiration,” Instacart chief architect JJ Zhuang told me via email. “What if AI could take on that mental load, and we could help the household leaders who are commonly responsible for grocery shopping, meal planning, and putting food on the table — and actually make grocery shopping truly fun? Instacart’s AI system, when integrated with OpenAI’s ChatGPT, will enable us to do exactly that, and we’re thrilled to start experimenting with what’s possible in the Instacart app.”

Those who’ve been closely following the ChatGPT saga, though, might be wondering if it’s ripe for release — and rightly so.

Early on, users were able to prompt ChatGPT to answer questions in racist and sexist ways, a reflection of the biased data on which ChatGPT was initially trained. (ChatGPT’s training data includes a broad swath of internet content, namely ebooks, Reddit posts and Wikipedia articles.) ChatGPT also invents facts without disclosing that it’s doing so, a phenomenon in AI known as hallucination.

ChatGPT — and systems like it — are susceptible to prompt-based attacks as well, or malicious adversarial prompts that get them to perform tasks that weren’t a part of their original objectives. Entire communities on Reddit have formed around finding ways to “jailbreak” ChatGPT and bypass any safeguards that OpenAI put in place. In one of the less offensive examples, a staffer at startup Scale AI was able to get ChatGPT to divulge information about its inner technical workings.

Brands, no doubt, wouldn’t want to be caught in the crosshairs. Brockman is adamant they won’t be. Why so? One reason, he says, is continued improvements on the backend — in some cases at the expense of Kenyan contract workers. But Brockman emphasized a new (and decidedly less controversial) approach that OpenAI calls Chat Markup Language, or ChatML. ChatML feeds text to the ChatGPT API as a sequence of messages together with metadata. That’s as opposed to the standard ChatGPT, which consumes raw text represented as a series of tokens. (The word “fantastic” would be split into the tokens “fan,” “tas,” and “tic,” for example.)

For example, given the prompt “What are some interesting party ideas for my 30th birthday?” a developer can choose to append that prompt with an additional prompt like “You are a fun conversational chatbot designed to help users with the questions they ask. You should answer truthfully and in a fun way!” or “You are a bot” before having the ChatGPT API process it. These instructions help to better tailor — and filter — the ChatGPT model’s responses, according to Brockman.

“We’re moving to a higher-level API. If you have a more structured way of representing input to the system, where you say, ‘this is from the developer’ or ‘this is from the user’ … I should expect that, as a developer, you actually can be more robust [using ChatML] against these kinds of prompt attacks,” Brockman said.

Another change that’ll (hopefully) prevent unintended ChatGPT behavior is more frequent model updates. With the release of gpt-3.5-turbo, developers will by default be automatically upgraded to OpenAI’s latest stable model, Brockman says, starting with gpt-3.5-turbo-0301 (released today). Developers will have the option to remain with an older model if they so choose, though, which might somewhat negate the benefit.

Whether they opt to update to the newest model or not, Brockman notes that some customers — mainly large enterprises with correspondingly large budgets — will have deeper control over system performance with the introduction of dedicated capacity plans. First detailed in documentation leaked earlier this month, OpenAI’s dedicated capacity plans, launched today, let customers pay for an allocation of compute infrastructure to run an OpenAI model — for example, gpt-3.5-turbo. (It’s Azure on the backend, by the way.)

In addition to “full control” over the instance’s load — normally, calls to the OpenAI API happen on shared compute resources — dedicated capacity gives customers the ability to enable features such as longer context limits. Context limits refer to the text that the model considers before generating additional text; longer context limits allow the model to “remember” more text essentially. While higher context limits might not solve all the bias and toxicity issues, they could lead models like gpt-3.5-turbo to hallucinate less.

Brockman says that dedicated capacity customers can expect gpt-3.5-turbo models with up to a 16k context window, meaning they can take in four times as many tokens as the standard ChatGPT model. That might let someone paste in pages and pages of tax code and get reasonable answers from the model, say — a feat that’s not possible today.

Brockman alluded to a general release in the future, but not anytime soon.

“The context windows are starting to creep up, and part of the reason that we’re dedicated-capacity-customers-only right now is because there’s a lot of performance tradeoffs on our side,” Brockman said. “We might eventually be able to offer an on-demand version of the same thing.”

Given OpenAI’s increasing pressure to turn a profit after a multibillion-dollar investment from Microsoft, that wouldn’t be terribly surprising.

OpenAI launches an API for ChatGPT, plus dedicated capacity for enterprise customers by Kyle Wiggers originally published on TechCrunch