Two thoughts.
One — anyone still using OpenAI without a subscription is getting third-tier AI results. GPT-4 has been reserved for paying ChatGPT Plus users since March, with a usage cap even for them, and through the API it costs $0.03 per 1,000 prompt tokens and $0.06 per 1,000 completion tokens. Set against the $0.002 per 1,000 tokens that gpt-3.5-turbo launched at, that’s fifteen to thirty times the price. The gap in the quality of the answers is why people pay it. Everyone on the free tier is judging AI by the older model.
Two — does anyone know of API access to the 70-billion-parameter, top-of-the-line open-source LLMs? Or is it worth us collectively establishing a service for them and provisioning the resource ourselves? Running a 70-billion-parameter model yourself means renting serious GPUs, and paying for them whether or not anyone is asking questions that hour.
The models exist. Meta released Llama 2 in July, “free for research and commercial use”, in 7-, 13- and 70-billion-parameter sizes. The hosting is starting to exist too. Anyscale opened an endpoints service last month serving Llama-2 70B at “$1 per million tokens,” which it says is “less than half the cost of comparable proprietary solutions for general workloads and up to 10X less expensive for specific tasks.” That’s half of gpt-3.5-turbo’s launch price for a 70-billion-parameter model. Whether it’s better for any given job is the kind of thing you only find out by running it.
Perhaps it’s also worth a beer-money business: establish third-party API access to the latest and greatest top models, and scale our own GPU access. A couple of startups — and our own — could split the cost, using gpt-3.5 for the drudgery and saving the big models for work that deserves them. None of us needs a whole GPU cluster. We need a reliable slice of one, at a price that lets the small shops use the big models too.
Johnson defined his own trade, lexicographer, as “a harmless drudge.” There’s a great deal of harmless drudgery in a startup, and gpt-3.5 is cheap enough to do most of it.
Raise the tide.
