Reading through OpenAI’s summary of GPT-4o today, what I find most interesting is something Sam Altman said in an interview on Friday: that algorithmic improvements to how LLMs are run could drastically reduce the compute needed for inference — the computation it takes to render a response — perhaps by enough to put a model like this locally on a phone.
Some context on what shipped. GPT-4o — the “o” is for “omni” — accepts “any combination of text, audio, image, and video” as input and answers in text, audio or image. It responds to speech in “as little as 232 milliseconds, with an average of 320 milliseconds”, which OpenAI says is about the pace of a person in conversation. The old Voice Mode took 2.8 seconds on GPT-3.5 and 5.4 on GPT-4, because it chained three separate models together. GPT-4o is a single model trained end to end across text, vision and audio. OpenAI says it matches GPT-4 Turbo on English text and code, “with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API.” For developers, “GPT-4o is 2x faster, half the price, and has 5x higher rate limits compared to GPT-4 Turbo.” And it’s going into ChatGPT’s free tier.
They made great strides this time around in foreign languages, and they also cut the computation needed to run in English by about 10%. That figure seems to be the tokenizer at work. GPT-4o has a more parsimonious one: the same short greeting that cost 145 tokens in Gujarati now costs 33, Hindi drops from 90 to 31, and English goes from 27 tokens to 24. Fewer tokens for the same sentence means less computation per answer, in every language, before anyone touches the model itself.
OpenAI’s own framing points the same way. It says it spent “a lot of effort over the last two years working on efficiency improvements at every layer of the stack”, and it calls GPT-4o “a first fruit of this research.”
I’m sure this process will continue. Imagine your eyeglasses having GPT-4o installed locally, to be your AI friend on the go.
