DeepSeek V3: New Open AI Model Surpasses Rivals and Challenges GPT-4o

Chinese artificial intelligence lab DeepSeek has introduced DeepSeek V3, its next genopen-source language model. Featuring 671 billion parameters, the model employs a so-called Mixture-of-Experts (MoE) architecture to combine computational efficiency with high performance.

DeepSeek V3’s technical advancements place it among the most powerful AI systems to, rivaling both open-source competitors like Meta’s Llama 3.1 and proprietary models like OpenAI’s GPT-4o.

The release highlights an important moment in AI, demonstrating that open-source systems can compete with—and in some cases outperform—costlier, closed alternatives.

Efficient and Innovative Architecture

DeepSeek V3’s architecture combines two advanced concepts to achieve exceptional efficiency and performance: Multi-Head Latent Attention (MLA) and Mixture-of-Experts (MoE).

MLA enhances the model’s ability to process complex inputs by using multiple attention heads to focus on different aspects of the data, extracting rich and diverse contextual information.

MoE, on the other hand, activates only a subset of the model’s total 671 billion parameters—approximately 37 billion per task—ensuring that computational resources are used effectively without compromising accuracy. Together, these mechanisms enable DeepSeek V3 to deliver high-quality outputs while reducing infrastructure demands.

Addressing common challenges in MoE systems, such as uneven workload distribution among experts, DeepSeek introduced an auxiliary-loss-free load-balancing strategy. This dynamic method allocates tasks across the network of experts, maintaining consistency and maximizing task accuracy.

Illustration of the basic architecture of DeepSeek-V3 (Image: DeepSeek)

To further enhance efficiency, DeepSeek V3 employs Multi-Token Prediction (MTP), a feature that allows the model to generate multiple tokens simultaneously, significantly accelerating text generation.

This feature not only improves training efficiency but also positions the model for faster real-world applications, reinforcing its standing as a leader in open-source AI innovation.

Benchmark Performance: A Leader in Math and Coding

DeepSeek V3’s benchmark results showcase its exceptional capabilities across a broad spectrum of tasks, solidifying its position as a leader among open-source AI models.

Leveraging its advanced architecture and extensive training dataset, the model has achieved top-tier performance in math, coding, and multilingual benchmarks, while also presenting competitive results in areas traditionally dominated by closed-source models like OpenAI’s GPT-4o and Anthropic’s Claude 3.5 Sonnet.

🚀 Introducing DeepSeek-V3!

Biggest leap forward yet:
⚡ 60 tokens/second (3x faster than V2!)
💪 Enhanced capabilities
🛠 API compatibility intact
🌍 Fully open-source models & papers

🐋 1/n pic.twitter.com/p1dV9gJ2Sd
— DeepSeek (@deepseek_ai) December 26, 2024

Mathematical Reasoning

On the Math-500 test, a benchmark designed to evaluate mathematical problem-solving skills, DeepSeek V3 achieved an impressive score of 90.2. This score places it ahead of all open-source competitors, with Qwen 2.5 scoring 80 and Llama 3.1 trailing at 73.8. Even GPT-4o, a closed-source model renowned for its general capabilities, scored slightly lower at 74.6. This performance underscores DeepSeek V3’s advanced reasoning abilities, particularly in computationally intensive tasks where precision and logic are critical.

Additionally, DeepSeek V3 excelled in other math-specific tests, such as:

MGSM (Math Grade School Math): Scored 79.8, surpassing Llama 3.1 (69.9) and Qwen 2.5 (76.2).
CMath (Chinese Math): Scored 90.7, outperforming both Llama 3.1 (77.3) and GPT-4o (84.5).
These results highlight its strength not only in English-based mathematical reasoning but also in tasks requiring language-specific numerical problem-solving.

Programming and Coding

DeepSeek V3 demonstrated remarkable prowess in coding and problem-solving benchmarks. On Codeforces, a competitive programming platform, the model achieved a 51.6 percentile ranking, reflecting its ability to handle complex algorithmic tasks. This performance significantly outpaces open-source rivals like Llama 3.1, which scored only 25.3, and even challenges Claude 3.5 Sonnet, which registered a lower percentile. The model’s success was further validated by its high scores in coding-specific benchmarks:

HumanEval-Mul: Scored 82.6, outperforming Qwen 2.5 (77.3) and matching GPT-4o (80.5).
LiveCodeBench (Pass@1): Scored 37.6, ahead of Llama 3.1 (30.1) and Claude 3.5 Sonnet (32.8).
CRUXEval-I: Scored 67.3, significantly better than both Qwen 2.5 (59.1) and Llama 3.1 (58.5).

These results highlight the model’s suitability for applications in software development and real-world coding environments, where efficient problem-solving and code generation are paramount.

Multilingual and Non-English Tasks

DeepSeek V3 also stands out in multilingual benchmarks, showcasing its ability to process and understand a wide array of languages. On the CMMLU (Chinese Multilingual Language Understanding) test, the model achieved an exceptional score of 88.8, surpassing Qwen 2.5 (89.5) and dominating Llama 3.1, which lagged behind at 73.7. Similarly, on C-Eval, a Chinese evaluation benchmark, DeepSeek V3 scored 90.1, well ahead of Llama 3.1 (72.5).

In non-English multilingual tasks:

MMMLU-non-English (Multilingual Massive Multitask Language Understanding): Scored 79.4, outperforming both Qwen 2.5 (74.8) and Llama 3.1 (73.8).
These results underline its capability to handle diverse languages effectively, making it a versatile tool for global AI applications.

English-Specific Benchmarks

While DeepSeek V3 excels in math, coding, and multilingual performance, its results in certain English-specific benchmarks reflect room for improvement. For instance, on the SimpleQA benchmark, which assesses a model’s ability to answer straightforward factual questions in English, DeepSeek V3 scored 24.9, falling behind GPT-4o, which achieved 38.2. Similarly, on FRAMES, a benchmark for understanding complex narrative structures, GPT-4o scored 80.5, compared to DeepSeek’s 73.3.

Despite these gaps, the model’s performance remains highly competitive, particularly given its open-source nature and cost efficiency. The slight underperformance in English-specific tasks is offset by its dominance in math and multilingual benchmarks, areas where it consistently challenges and often surpasses closed-source rivals.

DeepSeek V3’s benchmark results not only demonstrate its technical sophistication but also position it as a versatile, high-performing model for a wide range of tasks. Its superiority in math, coding, and multilingual benchmarks highlights its strengths, while its competitive results in English tasks show its ability to contend with industry leaders like GPT-4o and Claude 3.5 Sonnet.

By delivering these results at a fraction of the cost associated with proprietary systems, DeepSeek V3 illustrates the potential of open-source AI to rival—and in some cases outperform—closed-source alternatives.

Cost-Effective Training at Scale

One of the standout achievements of DeepSeek V3 is its cost-efficient training process. The model was trained on a dataset of 14.8 trillion tokens using Nvidia H800 GPUs, with a total training time of 2.788 million GPU hours. The overall cost amounted to $5.576 million, a fraction of the estimated $500 million required to train Meta’s Llama 3.1.

The NVIDIA H800 GPU is a modified version of the H100 GPU designed for the Chinese market to comply with export regulations. Both GPUs are based on NVIDIA’s Hopper architecture and are primarily used for AI and high-performance computing applications. The H800’s chip-to-chip data transfer rate is reduced to approximately half of the H100’s

The training process employed advanced methodologies, including FP8 mixed precision training. This approach reduces memory usage by encoding data in an 8-bit floating-point format without sacrificing accuracy. Additionally, the DualPipe algorithm optimized pipeline parallelism, ensuring smooth coordination across GPU clusters.

DeepSeek says that pre-training DeepSeek-V3 required only 180,000 H800 GPU hours per trillion tokens, using a cluster of 2,048 GPUs.

Accessibility and Deployment

DeepSeek has made V3 available under an MIT license, providing developers with access to the model for both research and commercial applications. Enterprises can integrate the model via the DeepSeek Chat platform or API, which is competitively priced at $0.27 per million input tokens and $1.10 per million output tokens.

The model’s versatility extends to its compatibility with various hardware platforms, including AMD GPUs and Huawei Ascend NPUs. This ensures broad accessibility for researchers and organizations with diverse infrastructure needs.

DeepSeek highlighted its focus on reliability and performance, stating, “To ensure SLO compliance and high throughput, we employ a dynamic redundancy strategy for experts during the prefilling stage, where high-load experts are periodically duplicated and rearranged for optimal performance.”

Broader Implications for the AI Ecosystem

DeepSeek V3’s release underscores a broader trend toward the democratization of AI. By delivering a high-performance model at a fraction of the cost associated with proprietary systems, DeepSeek is challenging the dominance of closed-source players like OpenAI and Anthropic. The availability of such advanced tools enables wider experimentation and innovation across industries.

DeepSeek’s pipeline incorporates verification and reflection patterns from its R1 model into DeepSeek-V3, improving reasoning capabilities while maintaining control over the output style and length.

The success of DeepSeek V3 raises questions about the future balance of power in the AI industry. As open-source models continue to close the gap with proprietary systems, they provide organizations with competitive alternatives that prioritize accessibility and cost-efficiency.