Groq.com — holy cow. Maybe I’m late to the game, but this is incredible: LLM response times in the milliseconds. I’ve been spouting on for a while about using LLMs not for word generation but as logic processors, and Groq makes that possible. Imagine making thousands of sequential LLM requests, each one dependent on the previous response, and getting the result back in under a second. You could run highly advanced reasoning applications at computer time.
Some context on the speed, because it’s the whole point. On February 13, Groq reported that ArtificialAnalysis.ai had independently benchmarked its Llama 2 Chat (70B) API “as achieving throughput of 241 tokens per second, more than double the speed of other hosting providers”. The hardware is Groq’s own LPU, which the company says is “architected from the ground up to achieve low latency”. SemiAnalysis notes that the demos going around run “the leading open-source model”, Mixtral 8x7B, at “up to 4x the throughput of other inference services while also charging less than 1/3 that of Mistral themselves.”
We’ve been doing these cascading processes at Ragu for a while now. This will supercharge both our current outputs and the kinds of challenges we take on.
It’s running mostly open-source models, and I think the trick is this: don’t use the LLM for word output, or for facts from its general knowledge base. Use it for simple, easily verified human-logic questions — and answers.
An example. Read this successful business’s exceptional business plan and prepare a question that is answered by every fact expressed in the document. Create as many as possible. A separate process takes those outputs and builds a question-and-answer list for the document. Now adapt every question in the list so it pertains to this other business idea instead — call it fashion. Now attempt to answer each of those refactored questions from data in this RAG set, perhaps including your company data and Gartner research. Your new question-and-answer set is now specific to your fashion business. Now create a detailed outline of the great fashion business plan. Now ask the LLM to draft every page in the outline using the answers from its question-and-answer run — not its crowd-wisdom knowledge set.
It’s perhaps not a great example, but notice that none of that process asked the LLM to use its training data. It used its human-logic capability to solve much smaller, verifiable tasks and built something bigger out of them. As the ask gets bigger, Groq’s speed at processing these logic requests becomes invaluable.
The whole approach above is a bet on knowing where to find the information rather than carrying it around.
And that is probably just the beginning. By looking at really big problems — fission, maybe — you could design ways for an LLM to use human logic to break a problem into all of its constituent parts, and then build test sequences to rapidly try to answer each question in turn.
I think we’re at the very start of understanding how to use this. But the point is that an LLM being right only part of the time on facts might just mean we’ve been using it wrong. At its core, its ability to reason through simpler human logic could be the real breakthrough. It’s certainly what we’ve been working on, now with a serious upgrade in our ability to imagine.
I’m not sure I know the answer yet, lol. Is language the product of human logic, or something more like physics? But I do think LLMs may be a new kind of operating system — one that is never going to be a reliable co-founder, but might be the excellent building blocks of one-millionth of an employee, doing very quick thinking on very simple thought tests. “Can you clearly tell who is the client and who is the salesperson in this email?” If yes, next step: “Who is the client, and who is the salesperson?”
