I ran off for a bit — honeymoon, which rocked — and I’ve only just had a chance to catch up. So this morning I spent two hours in conversation with Bard, which I believe was giving me Gemini Pro responses rather than Ultra. Then I pasted the same prompts into Claude and GPT-4 to judge their reactions to the same stimuli.
The Pro guess checks out. When Google announced Gemini on December 6, it said Bard “will use a fine-tuned version of Gemini Pro for more advanced reasoning, planning, understanding and more”, with Ultra held back for a “Bard Advanced” tier arriving “early next year.”
The conversation opened with 6–8K of context about the business I’m spinning up. I asked for a comparison between what we’re doing and the significant competitors. Then an investor presentation. Then we dived into a specific product and, finally, began designing product mockups together.
I was impressed with Gemini Pro — enough to consider porting much of my time in its direction, especially given features like export into Google Docs and its ability to examine documents in my Drive or my email exchanges. The Drive and email access arrived with Bard Extensions in September, which reach into “Gmail, Docs, Drive”. As I ran the comparison, it also clearly bested GPT-3.5 and Claude. I thought it was as good as GPT-4 — until I started actually pasting into GPT-4.
What became clear there was that I wanted a different conversation with GPT-4 than the one I’d been having with Gemini Pro. GPT-4 was discerning nuances and grasping my intent well enough that I’d have adjusted my inputs if I’d started with it. My user journey would have been further along, and more efficient. GPT-4 also offered actual mockups — images — when we got to that part. They weren’t great, but I may use aspects of them.
Bottom line: I’m not giving up my GPT-4 subscription, but I’ll be using Bard and Gemini a lot more often. On creativity in matters of design, Bard may have impressed me more, and I’d like to research other people’s impressions of “creativity” here.
Finally, I think I’ll give up my Claude 2 subscription. Anthropic’s pitch is the window — Claude 2 launched taking up to 100K tokens per prompt, enough that “Claude can work over hundreds of pages of technical documentation or even a book.” I’m not sure that “superior” context window is worth the inferior results anymore.
