Neuronpedia is an open-source research and engineering platform designed for mechanistic interpretability—the subfield of artificial intelligence safety and research focused on understanding, visualizing, and steering the internal representations of deep neural networks. Launched initially to accelerate research surrounding Sparse Autoencoders (SAEs), the platform has expanded into a generalized infrastructure powering white-box neural network analysis.
Neuronpedia hosts over 50 terabytes of activations, feature dashboards, explanations, and model metadata. By managing complex infrastructure tasks—such as scaling activation data storage, generating auto-interpretability explanations, and maintaining web-based interactive tooling—Neuronpedia allows AI safety researchers, red-teamers, and developers to explore and manipulate model internals without having to build custom visualization pipelines from scratch.
Core Technical Philosophy & Architecture
Modern large language models (LLMs) operate using dense vector representations, making individual neurons polysemantic (responding to multiple, unrelated concepts simultaneously). Mechanistic interpretability techniques like Sparse Autoencoders (SAEs), Probes, and Transcoders decompose these dense hidden states into monosemantic "features" or "latents"—interpretable units representing specific concepts, syntax patterns, or reasoning paths.
+----------------------------------------------+
| Raw Deep Neural Network / LLM |
| (Dense, Polysemantic Layer Activations) |
+----------------------+-----------------------+
|
v
+----------------------------------------------+
| Sparse Autoencoders (SAEs) |
| & Mechanistic Probes / Transcoders |
+----------------------+-----------------------+
|
v
+----------------------------------------------+
| Neuronpedia Platform |
| - 90,000,000+ Monosemantic Latents Indexed |
| - Automated Feature Explanations & Scoring |
| - Live Interventions & Activation Testing |
+----------------------+-----------------------+
|
+-----------------+-----------------+
| |
v v
+----------------------------+ +----------------------------+
| Feature Dashboards & Web UI| | OpenAPI / Python / TS API |
| (Steering, Circuit Tracer) | | (Export, Batch Evaluation) |
+----------------------------+ +----------------------------+
Neuronpedia acts as a centralized repository and interactive laboratory for these extracted features across dozens of open-weights models, including families like GPT-2, Gemma, Llama, and Qwen.
Core Products and Platform Tools
Neuronpedia offers a suite of interactive web interfaces and programmatically accessible tools designed for feature discovery, concept search, circuit visualization, and live model steering.
1. Interactive Feature Dashboards & Explanations
For every indexed Sparse Autoencoder or probe layer, Neuronpedia automatically generates an interactive dashboard.
* Top Activations & Logits: Displays the specific text tokens that trigger a given feature, alongside positive and negative logit influences (what words the feature pushes the model to predict next).
* Automated Interpretation (Autointerp): Utilizes language models (such as GPT-4 or Claude) to automatically summarize what concept a feature represents, providing a confidence/explanation score.
* Live Inference Testing: Allows researchers to input custom text prompts into a live feature dashboard to verify whether a specific feature activates as hypothesized.
2. Feature Search Engines
Neuronpedia indexes over 90 million latents and features, searchable through two primary methods:
* Search by Explanation: Enables natural language search across feature summaries to locate concepts (e.g., finding features associated with "Star Wars," "deception detection," or "code syntax errors").
* Search via Inference: Allows users to feed custom text through an inference pass to identify which specific internal features across layers fire most strongly.
3. Activation Steering (SteerSAE)
Neuronpedia provides a live workspace to perform activation addition and feature steering. Users can alter a model's generation behavior in real time by increasing or decreasing the strength (clamp coefficient) of specific features or custom vectors during generation. This allows researchers to test behavioral shifts—such as amplifying honesty, inducing persona drifts, or suppressing specific topics—without retraining the underlying model.
4. Circuit Tracer & Attribution Graphs
Building on circuit analysis methodologies pioneered by leading research labs, the Circuit Tracer tool visualizes the internal computation graphs of models. Users can trace how information flows from input tokens through attention heads, MLP layers, transcoders, and output logits. Custom attribution graphs can be uploaded and rendered directly within the platform interface.
5. Natural Language Autoencoders (NLAs) & Advanced Interfaces
Neuronpedia regularly integrates cutting-edge interpretability primitives, such as Natural Language Autoencoders (NLAs). This feature translates raw model activation snapshots directly into human-readable text logs, allowing researchers to monitor real-time "thoughts" or hidden reasoning steps during tasks involving deception, persona drift, or prompt injection attacks.
Developer Services and Infrastructure
Interpretability API & SDKs
Neuronpedia provides an open-access API along with official client libraries in Python and TypeScript. The API enables programmatic interaction with all platform functionalities, including:
* Querying feature activation histories, token distributions, and logit weights.
* Triggering live inference calls with modified activation vectors.
* Uploading custom-trained Sparse Autoencoders, probes, or attribution graphs to auto-generate hosted public or private dashboards.
Data Hosting and Collaboration
By providing hosted storage for terabytes of activation sets and sparse representations, Neuronpedia functions as a collaborative registry for the AI safety community. Researchers and institutions can host their trained SAE artifacts (such as DeepMind's GemmaScope sets) on Neuronpedia, making them instantly browsable, shareable, and embeddable via web iframe components.
Primary Use Cases & Applications
- AI Safety & Alignment: Identifying where and how safety-critical concepts (e.g., deception, bias, power-seeking behavior) are represented within model hidden states.
- Red-Teaming & Vulnerability Discovery: Uncovering hidden failure modes or backdoors by searching for latent features triggered by specific prompt injection techniques.
- Mechanistic Model Editing: Utilizing targeted activation steering to suppress unsafe behaviors or fine-tune model outputs without modifying parameter weights.
