DeepSeek LLM

DeepSeek LLM is the first general-purpose language model series released by DeepSeek, the Chinese AI lab spun out of the quantitative hedge fund High-Flyer. Announced in November 2023 with base and chat variants at 7 billion and 67 billion parameters, it established the open-weight release pattern that later entries in the DeepSeek family would follow, and its accompanying technical report (arXiv:2401.02954, January 2024) framed the lab's long-horizon research agenda under the phrase "scaling open-source language models with longtermism."¹

Design and training

Architecturally, DeepSeek LLM is a dense decoder-only Transformer that closely follows the LLaMA recipe: pre-normalization with RMSNorm, SwiGLU feed-forward layers, and rotary position embeddings, with the 67B model adopting grouped-query attention to reduce inference cost. Both sizes were trained on a developer-reported 2 trillion tokens of bilingual English and Chinese data, with a 4,096-token context window.¹

The report devoted substantial attention to Scaling laws, revisiting the compute-optimal analysis associated with Chinchilla and arguing that optimal model and data allocation depends measurably on data quality. Rather than fixing hyperparameters per model size, the team fit scaling behavior for batch size and learning rate, an empirical approach that carried into later DeepSeek work.²

Chat variants and evaluation

The chat models were produced through supervised instruction tuning followed by direct preference optimization rather than a full RLHF pipeline. In developer-reported evaluations, DeepSeek LLM 67B Chat outperformed Llama 2 70B on benchmarks including code, mathematics, and reasoning, and compared favorably with GPT-3.5 on open-ended Chinese and English generation, while base-model scores on MMLU were competitive with contemporaneous open models such as Qwen.¹

Legacy

DeepSeek LLM was quickly superseded within its own family: DeepSeek-V2 moved the lab to a mixture-of-experts architecture in 2024,³ and DeepSeek-V3 and DeepSeek-R1 brought the lab global attention. The series nonetheless remains notable as the origin point of DeepSeek's open-weight strategy and of the scaling-law methodology the lab reused in subsequent reports.²

References