Artificial intelligence (AI) startup Anthropic has announced the release of Claude 2, their second generation of its AI chatbot. The underlying new model brings improved performance, and longer responses, and is accessible via API as well as a new public-facing beta website, claude.ai.
Enhanced Performance and Capabilities
Claude 2 has shown significant improvements in coding, math, and reasoning compared to its predecessor. The AI model scored 76.5% on the multiple-choice section of the Bar exam, up from 73.0% with Claude 1.3.
The Bar exam is a standardized test that evaluates the legal knowledge and skills of candidates who want to practice law in a certain jurisdiction. It is usually composed of multiple-choice questions, essays, and performance tests that cover various areas of law, such as constitutional law, criminal law, contracts, evidence, and ethics. AI systems that can pass the Bar exam are expected to demonstrate a high level of proficiency in reading comprehension, logical reasoning, argumentation, and writing. Some of the challenges that AI systems face in taking the Bar exam include dealing with complex and ambiguous legal scenarios, applying relevant laws and precedents, and producing coherent and persuasive answers.
In terms of coding skills, Claude 2 scored a 71.2% up from 56.0% on the Codex HumanEval, a Python coding test. The new model can handle longer input and output, analyzing documents of up to 100,000 tokens and writing content with up to a few thousand tokens. This allows Claude to work over hundreds of pages of technical documentation or even a book.
Anthropic has emphasized safety in developing Claude 2, making it less likely to generate harmful content. The company uses extensive techniques and testing to reduce risks. In an internal red-teaming evaluation that scores models on a large representative set of harmful prompts, Claude 2 was twice as good at giving harmless responses compared to Claude 1.3.
Asked about the difference between Claude 2, Google Bard, ChatGPT and Bing Chat, the AI chatbot gave the following, quite limited answer:
-
-
- – I am Claude, an AI assistant created by Anthropic. I was designed with self-consistency and harm avoidance in mind.
- – Google Bard is an experimental conversational AI service launched by Google in February 2023. It is designed to provide helpful information to users’ questions.
- – ChatGPT is an AI chatbot created by OpenAI and launched in November 2022. It is designed to have natural conversations and provide human-like responses.
- – Bing Chat is Microsoft’s AI chatbot integrated into the Bing search engine in February 2023. It aims to provide more conversational and comprehensive search responses.
The main differences come down to the companies behind us, our capabilities, and how we were designed. I was designed with safety in mind, while the others aim to provide the most informative responses, even if inconsistent at times. ChatGPT also led the wave of releasing generative AI to the public. Ultimately we have similarities in being AI assistants but take different approaches based on our creators’ priorities.”
- – I am Claude, an AI assistant created by Anthropic. I was designed with self-consistency and harm avoidance in mind.
-
Availability and Future Plans
Claude 2 is currently available to users in the U.S. and U.K., with plans to expand availability in the coming months. The Claude 2 API for businesses is being offered for the same price as Claude 1.3. Anthropic has an exciting roadmap of capability improvements planned for Claude 2 and will be slowly and iteratively deploying them in the coming months.
Anthropic is currently working with thousands of businesses who are using the Claude API. Early partners, including Jasper and Sourcegraph, are already integrating the new version of Claude via the API and finding value in its enhanced semantics, reasoning, and larger context window.
User Feedback and Responsible Deployment
Anthropic welcomes user feedback as they work to responsibly deploy their products more broadly. The company acknowledges that Claude – like all current models – can generate inappropriate responses and should not be used where physical or mental health and well-being are involved.
About Anthropic
Anthropic is an American artificial intelligence (AI) startup and public-benefit corporation, founded by former members of OpenAI. The company was established in 2021 by siblings Daniela Amodei and Dario Amodei, the latter of whom served as OpenAI’s Vice President of Research. The founders were among others who left OpenAI due to directional differences, specifically regarding OpenAI’s ventures with Microsoft in 2019.
Anthropic specializes in developing general AI systems and language models, with a company ethos of responsible AI usage. As of July 2023, Anthropic had raised US$1.5 billion in funding. The company has also faced legal challenges, such as a lawsuit in February 2023 by Texas-based Anthrop LLC for the use of their registered trademark “Anthropic A.I.” In addition to its AI chatbot Claude, Anthropic also publishes research on the interpretability of machine learning systems, focusing on transformer architecture.
