Sign up¹⁴⁸ | Advertise⁹⁴ | Ben’s Bites News⁸³
Daily Digest #301

Hello folks, here’s what we have today;

PICKS

1. I wrote a business deep-dive on Replicate, the AI platform enabling entrepreneurs to make millions per year.^2,142 They’re valued at $350M and I did some real digging into the early days and how they got to where there they are.

2. Google launched Gemini out of the blue yesterday.^1,016 Gemini is trained on text, images, and audio together. It comes in three sizes: Ultra (beats GPT-4), Pro (powers Bard now) and Nano (for mobile devices).

   In the benchmarks, all I see are blue numbers under Gemini with GPT-4 (and GPT-4 Vision) greyed out on the side. Google has proved that a model beating GPT-4 is possible but again, ships a (kinda) waitlist.🍿The balanced take you need⁹⁸¹ (also below)

3. Sam Altman is TIME’s CEO of the year.⁴⁵⁹ They made a new category, because Taylor Swift is the Person of the Year.

4. Meta has released a standalone tool for Emu⁶²⁹—its image generation models.

5. PS: highlighting again. Elon Musk has raised $135M for xAI⁴⁰¹ and wants to go up to $1B.

from our sponsor

Deploy LLM-powered features to production with confidence

Vellum²⁴⁴ provides tools for comparing prompts and models, managing complex LLM chains, quantitative testing, version control and performance monitoring.

Compatible across all major LLM providers like OpenAI, Anthropic, Google, Azure, AWS, Mistral, Llama and more.

Ben's Bites readers get 20% discount. Request a personalized demo here.²⁴⁴

TOP TOOLS

• n8n⁵⁰⁹* - A low-code platform for technical users to manage complex workflows & AI apps. Self-hostable, source-available, and flexible.

• Simple Analytics AI³⁷² - Chat with your website analytics.

• Mage⁴⁶⁵ - Use AI with vision in your Chrome sidebar.

• Streak³⁹³ - The copilot for your CRM.

• Djay by Algoriddim²²⁸ - Real-time stem separation for performing DJs. Powered by Audioshake.⁸⁵

• Music AI³¹⁴ - Collection of SOTA music APIs and AI audio solutions in a single platform.

• Seamless Expressive demo by Meta^257† - Speech-to-Speech translation that preserves emotion.

• Santa Cat⁴⁹⁶ - Been naughty? or nice? Give Santa Cat a call no matter what.

View more →¹⁷³
*sponsored

NEWS

• The one thing Google got wrong⁶³⁷ with Gemini (not benchmarks).

• Long context prompting for Claude 2.1²⁷⁶ - TLDR: Just add “Here’s the most relevant sentence in the context:” to your prompt.

• How to create an AI narrator⁴⁴¹ for your life.

• Liquid AI has raised a $37.6M seed round¹⁸⁷ for making AI models from first principles.

• 5-month-old startup Sarvam AI raises $41M seed¹⁴⁵ for building foundational models in Indian languages.

• Why a vertical approach³⁴⁷ is key to building enduring AI applications.

• All the talks from 2nd Cerebral Valley AI Summit.²⁰³

• Rumor: ByteDance has a “powerful than Gemini” model soon to be released.

View more →⁷²

QUICK BITES

Google launched Gemini out of the blue yesterday.^1,016 Well, not that out of the blue—The Information first reported that Google is postponing this launch to January and then updated the report to say nope, Google’s gonna do it this week.

And Google launched, with a bang.

I click on the announcement post and all I see are blue numbers under Gemini with GPT-4 (and GPT-4 Vision) greyed out on the side. Impressive stuff from Google. (TLDR at the bottom)

This looks pretty bad, right?

But for whom…

As the demo-induced excitement takes some rest, I (and the rest of AI Twitter) dive into the post and find some troubling things.

• The MMLU performance which Google claims surpasses GPT-4 (and even humans) is a clever play on prompting technique, Otherwise, Gemini is still losing to GPT-4. Almost near GPT-4, but losing.

• The flagship demo is a post-processed video (expected) with prompts read in the video different from actual prompts to the system (unexpected). Google reveals this on its own, by releasing a dev post “How it’s made’ breaking down how they created the video.

• The blue numbers are for Gemini Ultra, which is gonna come next year. The model is live right now in Bard is Pro, one version down. The developer access for even Pro models is a week away (13th December).

What exactly did Google launch then?

Let’s take a deep breath and think step by step.😏

The biggest breakthrough in the Gemini announcements is the fact that Gemini models are trained on multimodal data from the ground up.⁸⁵ This includes text, images, videos and audio. This change might result in Google getting a lead and OpenAI play catch up in 2024.

Gemini comes in three sizes: Ultra, Pro and Nano. Ultra beats GPT-4 on many benchmarks and is comparable on the rest. But we are not getting Ultra anytime soon. We’re getting Pro in Bard⁹², starting now. The kids in this party of giants, the Nano models will run on mobile devices⁷⁴ starting with the Pixel 8 Pro.

Gemini Pro will be in developers’ hands next week. Android devs will also get access to Gemini Nano. Gemini Ultra’s first appearance next year would be in a different product called Bard Advanced. It’ll likely combine these features and be paid.

The highlight of Benchmark performance is MMLU beating GPT-4 and Humans. They use a new prompting + reward technique to get to 90% on MMLU [technical report^111†]

Google's performance on vision and audio benchmarks is more impressive. Pro with Vision is comparable to GPT-4 with Vision, and Gemini wins over Whisper by a huge margin. But as we all know those are benchmarks. We tried replicating Gemini tests in ChatGPT⁶⁸ and felt GPT-4 did as well as the demo videos.

Ah! The demos. Tell me more

The key demo, Hands-on with Gemini¹⁴⁷, shows Gemini using image and audio inputs, working in multiple languages, writing code, and reasoning using images or videos as context. Obviously, the demo is cherry-picked and sped up with post-production (like audio outputs). Google’s behind-the-scenes article¹¹⁸ explains how much.

But there are about a dozen other demos buried below this one. A quick recap of the interesting ones:

• Gemini allows scientists to scan through 200,000 papers¹³⁰, find ~250 relevant ones and extract data from those papers.

• A special version of Gemini, AlphaCode2¹⁰⁸ performs better than 85% of humans in competitive programming.

• It can check your kids’s science homework⁹² or help you in listening to French podcasts.¹³⁸

• Gemini can create UIs on the fly.²³³ BIG!! They are calling it Bespoke UI in the demo. It’ll be interesting to see if this comes out as a product early next year.

So what?

The new suite of Gemini models is impressive. Google proved a model beating GPT-4 is possible but again, ships a waitlist. Gemini’s integration into Google products remains to be seen.

Bard with Gemini Pro is likely better than the ChatGPT free version. ChatGPT Plus with GPT-4 or Bing in Creative mode (using GPT-4 under the hood) is still better.

That’s it, the rest of it is drama.

Share this story⁹⁸¹

Ben’s Bites Insights

We have 2 databases that are updated daily which you can access by sharing Ben’s Bites using the link below;

• All 10k+ links we’ve covered, easily filterable (1 referral)

• 6k+ AI company funding rounds from Jan 2022, including investors, amounts, stage etc (3 referrals)