Elon Musk's company, xAI, has announced the release of two new beta models, Grok-2 and Grok-2 Mini. Available exclusively to Premium and Premium+ users on the X social network, these models offer sophisticated reasoning and image generation capabilities. Musk suggested in July that Grok-2 would launch soon, while also saying Grok-3 will also drop this year.
Capabilities and Features
Grok-2 is a significant upgrade over Grok-1.5, incorporating enhanced chat, coding, and reasoning functionalities. Grok-2 Mini delivers similar benefits in a more compact format. An early version of Grok-2, tested under the alias “sus-column-r,” outperformed models like Claude 3.5 Sonnet and GPT-4-Turbo on the LMSYS leaderboard.
The LMSYS Leaderboard is a platform that ranks large language models (LLMs) based on their performance in various tasks. It's essentially a competitive arena where different AI models face off against each other.
Performance and Evaluation
Both models have been thoroughly assessed across a range of academic benchmarks, such as reasoning, reading comprehension, math, science, and coding. Grok-2 and Grok-2 Mini surpass their predecessor in these areas, demonstrating strong performance in tasks related to graduate-level science (GPQA), general knowledge (MMLU, MMLU-Pro), and math competition problems (MATH). Additionally, Grok-2 excels in vision-based tasks like visual math reasoning (MathVista) and document-based question answering (DocVQA).
Premium users on the X platform can access Grok-2 and Grok-2 Mini via the Grok tab in the X app. Grok-2 combines real-time data from the X platform, enhancing text and vision understanding capabilities. The social media company is currently facing regulatory pushback for its use of user data to train xAI models. Grok-2 Mini balances speed and response quality, catering to users who need efficient and accurate answers.
Developer Access and API
Later this month, xAI will provide developers with access to Grok-2 and Grok-2 Mini through its enterprise API. The platform will include multi-region inference deployments for low-latency access, advanced security features, detailed traffic statistics, and sophisticated billing analytics. A management API will also be available, facilitating the integration of user, team, and billing management into existing tools.
The initial version of Grok faced issues such as spreading unverified claims and inaccuracies in news summaries. Without proper safeguards, Grok-2 could be misused to create misinformation on X and other platforms. xAI has not yet answered questions regarding measures to prevent harmful uses of its image generation feature.
