Llemma: An Open Language Model and Infrastructure for Mathematics
Executive Overview
Llemma is an open-source artificial intelligence initiative and domain-specific language model suite designed specifically for mathematical reasoning, computation, and theorem proving. Developed as a joint research project by researchers across Princeton University, EleutherAI, the University of Toronto, Vector Institute, the University of Cambridge, Carnegie Mellon University, and the University of Washington, Llemma was introduced in the paper Llemma: An Open Language Model for Mathematics (arXiv:2310.10631) and published as a conference paper at ICLR 2024.
Unlike general-purpose base language models that split attention across all domains of natural language, Llemma is explicitly optimized for mathematical reasoning. By performing continued pretraining on specialized mathematical web pages, academic literature, and math-focused source code, Llemma achieves state-of-the-art mathematical capability among open base models. The project releases all underlying artifacts—including model weights, training corpora, and code—to serve as an open research platform for the global AI community.
Continue…
Core Operational Purpose and Mission
The primary purpose of the Llemma project is to advance machine intelligence in quantitative reasoning, symbolic manipulation, and formal verification. Llemma addresses a key bottleneck in AI research: prior state-of-the-art math models (such as Google's Minerva) were closed-access proprietary systems, restricting community analysis, fine-tuning, and reproduction.
Llemma provides an open foundation that bridges natural language, computational code, and formal logic. Its core activities encompass:
- Domain Adaptation at Scale: Continuing the pretraining of foundational code models on massive mathematical corpora.
- Multi-Modal Mathematical Reasoning: Interleaving natural text, LaTeX equations, computer algebra code, and formal theorem prover code.
- Open-Access AI Infrastructure: Publishing open-weights models and dataset artifacts to accelerate upstream AI research topics such as reward modeling, reinforcement learning for reasoning, and algorithmic synthesis.
Products and Model Offerings
The Llemma ecosystem consists of several primary open-source AI model artifacts released under open licenses for research and commercial development.
1. Llemma-7B
- Parameter Count: 7 billion parameters.
- Base Architecture: Initialized from Code Llama 7B (a decoder-only Transformer) and pretrained for 200 billion additional tokens on Proof-Pile-2.
- Training Footprint: Trained across 256 NVIDIA A100 (40GB) GPUs using GPT-NeoX, Tensor Parallelism, and FlashAttention-2.
- Use Cases: Efficient deployment for real-time mathematical problem solving, lightweight theorem proving assistance, and resource-constrained research environments.
2. Llemma-34B
- Parameter Count: 34 billion parameters.
- Base Architecture: Initialized from Code Llama 34B and pretrained for 50 billion additional tokens on Proof-Pile-2.
- Performance Profile: Outperforms all public open base models on challenging mathematical benchmarks (such as MATH) and beats the equi-parameter closed-source Minerva 8B and 62B models on key metrics.
- Use Cases: Advanced multi-step mathematical problem solving, complex code generation for numerical analysis, and formal theorem step generation.
Datasets and Open Data Assets
In addition to neural network weights, Llemma offers major datasets created specifically to support mathematical domain adaptation.
1. Proof-Pile-2 (55-Billion-Token Corpus)
Proof-Pile-2 is a comprehensive 55-billion-token training dataset compiled to train models in mathematical reasoning. It is made up of three primary pillars:
- AlgebraicStack (11 Billion Tokens): A specialized dataset of source code spanning 17 programming languages focused on numerical, symbolic, and formal mathematics. It includes code from Python, Julia, C++, Lean, Coq, Isabelle, Macaulay2, Sage, and Mathematica.
- OpenWebMath (15 Billion Tokens): High-quality web text extracted from Common Crawl, rigorously filtered for mathematical content and structured to preserve mathematical LaTeX formatting.
- ArXiv Subset (29 Billion Tokens): Academic scientific papers and preprints sourced from the RedPajama ArXiv subset, covering physics, computer science, mathematics, and quantitative biology.
2. General Domain Regularization Data
To prevent catastrophic forgetting during continued pretraining, the data pipeline mixes Proof-Pile-2 (95%) with general-domain text from The Pile (2%) and general code repositories from RedPajama GitHub (3%).
Technical Capabilities and Functional Services
Llemma delivers specialized capabilities across three distinct modes of mathematical processing:
1. Chain-of-Thought (CoT) Mathematical Reasoning
Llemma generates self-contained step-by-step LaTeX text solutions to complex mathematical problems.
* Multi-Step Deductive Execution: The model automatically identifies mathematical identities, swaps summation order, applies algebraic transformations, and performs telescoping sum simplifications.
* Benchmark Standard Performance: Solves competition-level math problems from high-school math competitions (MATH), middle-school word problems (GSM8k), MIT OpenCourseWare STEM courses (OCWCourses), and College Board SAT exams.
2. Computational Tool Integration (Program-Aided Math)
Llemma interacts directly with external computational tool environments without requiring supervised fine-tuning.
* Python Interpreter Execution: Generates Python scripts utilizing SymPy, NumPy, and SciPy to perform exact numerical calculations, solve differential equations, and evaluate complex matrix operations.
* Interleaved Tool Calling: Interleaves natural language reasoning steps with code execution blocks to verify intermediate values and eliminate arithmetic errors.
3. Formal Theorem Proving
Llemma functions as an assistant and auto-informant inside formal interactive theorem provers (ITPs).
* Lean, Coq, and Isabelle Support: Generates formal proof steps (tactics) in interactive systems such as Lean 3, Lean 4, Coq, and Isabelle.
* Automated Proof Search: Integrates with search algorithms (such as best-first search) to assist in formalizing unproven mathematical claims and verifying formal proofs automatically.
Key Performance Indicators and Benchmarks
Llemma demonstrates major performance advances over general-purpose open models (such as LLaMA-2 and Code Llama) and closed baselines (such as Minerva):
| Benchmark | Target Task Level | Performance Highlight |
| --- | --- | --- |
| MATH | High-School Competition Mathematics | Llemma-34B achieves top 4-shot accuracy among open base models, surpassing Minerva 8B and 62B equi-parameter runs. |
| GSM8k | Grade School Mathematical Word Problems | High few-shot CoT baseline accuracy without specialized instruction-tuning. |
| MMLU-STEM | Multi-discipline STEM Knowledge | Evaluates knowledge across 18 undergraduate STEM fields. |
| Tool-Use Verification | Python SymPy / Lean Theorem Proving | Strong ability to construct syntax-valid code for formal solvers and CAS interpreters. |
Infrastructure, Licensing, and Ecosystem Impact
- Open-Source Availability: All code, dataset pipelines, model checkpoints, and evaluation harnesses are hosted on GitHub (
EleutherAI/math-lm) and Hugging Face (EleutherAI/proof-pile-2, EleutherAI/llemma_7b, EleutherAI/llemma_34b).
- Research Platform Foundation: Llemma serves as a public foundation for downstream AI research including process-supervised reward modeling (PRMs), reinforcement learning from human/mathematical feedback (RLHF/RLAIF), and automated mathematical discovery.
