Super awesome updates today, in my opinion. Of everything OpenAI announced at its first DevDay, I’m most excited about two things: the extraordinary new size of the context window, and the Assistants API.
I should explain why those two. I’ve got my own personal assistant. It’s built on a RAG infrastructure and it lives in my own tenant on Amazon — think months of research and ultra-secure, expensive infrastructure. With these two updates, my AI PA will have even more superpowers.
Start with the window. GPT-4 Turbo takes 128,000 tokens of context — four times what GPT-4 could hold, or, as TechCrunch worked it out, “around 100,000 words or 300 pages.” It’s also cheaper: $0.01 per 1,000 input tokens and $0.03 per 1,000 output tokens, a third and a half of what GPT-4 charges.
For a RAG system, the window is the whole game. Retrieval’s job is to put the right documents in front of the model, and the window decides how many of them fit. Four times the room at a third of the input price should mean my assistant can see a great deal more of my own material per question.
Then the Assistants API. OpenAI describes it as a way to build an “assistant” that “has specific instructions, leverages outside knowledge and can call OpenAI generative AI models and tools to perform tasks,” with a retrieval component for a company’s own documents. It’s in beta and open to all developers from today.
We are heavy into commercialization. We’re even planning to publish the setup for comment.
