Earlier this week, Google took the wrapper off its new AI model. Known as Gemini, the platform is available in three tiers and is meant to position Google against OpenAI and Microsoft in the increasingly competitive AI market. While Google has boasted about Gemini's advanced architecture and its equal or superior performance to other notable AI models, including OpenAI's GPT-4, user experiences shared online paint a less favorable picture.
Inconsistencies in Simple Queries Raising Eyebrows
A critical component touted by Google is Gemini Pro's factual accuracy. However, users are taking to social media to share discrepancies in the AI's responses. A notable error includes providing incorrect information about the 2023 Oscar winners. Despite claims of improved reasoning and understanding over its predecessors, Gemini Pro attributed the Best Actor award to Brendan Gleeson instead of the actual recipient, Brendan Fraser. Similarly, missteps occurred when asked about other Oscar category winners.
Additionally, translation capabilities seem to be a weakness for Gemini Pro, with users reporting that asks for simple translations are met with incorrect responses. For instance, a request for a six-letter word in French was met with a reply that included either a five or seven-letter word, highlighting concerns about Gemini Pro's multilingual performance.
I'm extremely disappointed with Gemini Pro on Bard. It still give very, very bad results to questions that shouldn't be hard anymore with RAG.
A simple question like this with a simple answer like this, and it still got it WRONG. pic.twitter.com/5GowXtscRU
— Vitor de Lucca 🏳️🌈 (@vitor_dlucca) December 7, 2023
Coding Capabilities Questioned Amid Competitive AI Landscape
Google emphasized Gemini's coding prowess, yet users experimenting with the new AI model have reported issues. These include Gemini Pro's inability to correctly execute basic coding tasks, like writing the intersection of two polygons in Python—a problem that seems to affect not just Gemini but other AI models as well.
Tried gemini based Bard, and well, it still can't write intersection of two polygons. It's one of those rare relatively simple to express functions that wasn't ever implemented in python, there is no stack overflow post, and all these models fail on it. pic.twitter.com/RKjmkEw2Qr
— Filip Piekniewski🌻 🐘:@[email protected] (@filippie509) December 6, 2023
Comparisons are being made to GPT-3.5, the predecessor of GPT-4, highlighting Gemini Pro's challenges in areas like developing code for analog clocks and games like Tic Tac Toe, where it lags behind in terms of output quality and accuracy of results.
Furthermore, Gemini Pro's approach to sensitive queries is to encourage users to conduct their own searches, a stark contrast to the detailed summaries given by ChatGPT. This cautious stance can frustrate users looking for quick, in-depth information.
Despite these reported shortcomings, Gemini Pro is not the final or most capable version of Gemini, as Google plans to release Gemini Ultra next year, which may address some of the current concerns.
FYI, Google Gemini is complete trash. pic.twitter.com/EfNzTa5qas
— Benjamin Netter (@benjaminnetter) December 6, 2023
As advances in generative AI models progress rapidly, Google's commitment to enhancing Gemini's performance remains paramount to meet user expectations and stay competitive. The technology giant has yet to publicly respond to the criticisms, and it is unclear how quickly they can or will address the feedback from Gemini Pro's early users.
Did Google Launch Gemini Too Early?
We have been here before with Google AI products. In February, the company was blindsided by Microsoft's all-on approach to AI with the launch of Bing Chat. While Bard was already under development, Google decided to rush the AI chatbot to market. The result was an embarrassing showcase of the AI's capabilities. Bard launched a month after the showcase and was buggy. It has since improved greatly, but that early launch was rough.
It seems Gemini is going through a similar process. Interestingly, there were reports recently that Google would delay the launch of Gemini. Inaccuracies and issues with multi-lingual capabilities were said to be behind the delay. Of course, the postponement never happened, but concerns about the performance of Gemini are now unfolding post launch.
