1,445 GitHub stars in two weeks for a research toolkit with zero marketing spend. That tells you something about how hungry the ML community is for tools that actually explain what language models are doing inside. Anthropic just open-sourced the Jacobian Lens, a reference implementation of their groundbreaking 'Verbalizable Representations Form a Global Workspace in Language Models' paper. The tool lets anyone probe what a transformer model is thinking at every single layer, revealing the internal monologue that leads to the final output we see.
For AI founders, this matters more than most open-source releases this year. Interpretability is the single biggest unsolved problem in AI safety, and the Jacobian Lens is one of the first practical, production-friendly tools that makes it accessible to anyone with a GPU and a HuggingFace model.
What the Jacobian Lens Actually Does
The core idea is deceptively simple. A language model processes input through dozens of layers. At each layer, it maintains a 'residual stream' -- an evolving vector representation that gradually becomes more specific to the next token it will predict. The Jacobian Lens reads out what any intermediate layer's activation is disposed to make the model say. It linearly transports a residual-stream vector at any layer and position into the final-layer basis, then decodes it with the model's own unembedding into a ranked list of vocabulary tokens.
In plain English: it shows you what the model is 'thinking about' at the middle layers, not just the final output. The researchers provide a striking example: when the model sees the ASCII face character '^' (the nose position), mid-layer readings output the word 'nose' with high probability, even though the word 'nose' never appeared anywhere in the prompt. The model is internally representing facial geometry from ASCII characters, and the Jacobian Lens makes that visible for the first time.
The tool renders interactive layer-by-position visualizations. Each cell shows the lens top-1 word at that (position, layer) coordinate with a rank superscript over the full vocabulary. Click a cell to pin its top-1 token and get rank-tracking charts and a rank heatmap. The bottom row (L = n_layers - 1) is the model's actual output, so you can compare the internal thinking against the final prediction.
The Global Workspace Theory Behind It
The paper behind the Jacobian Lens -- 'Verbalizable Representations Form a Global Workspace in Language Models' -- is published on Anthropic's transformer-circuits.pub domain. It builds on the Global Workspace Theory from cognitive science, which proposes that conscious thought operates through a 'global workspace' where information from specialized processors is integrated and broadcast back. The paper demonstrates that verbalizable representations in language models behave like a global workspace: they integrate information across layers, are broadly accessible, and correlate with the model's confidence in its outputs.
The mathematical machinery is elegant. The lens is the average input-output Jacobian over a text corpus: J_l(h) = unembed( J_l @ h ), where J_l = E[dfinal / dh_l]. The expectation is over prompts, source positions, and all current-and-future target positions in a generic web-text corpus. The paper's lenses use 1,000 sequences of 128 tokens from a pretraining-like corpus, and quality saturates quickly -- the authors note that roughly 100 prompts is already usable.
How to Install and Use It
The Jacobian Lens requires Python and a CUDA-capable GPU. Installation is straightforward: pip install -e . from the cloned repository. The repository is published under the Apache 2.0 license, meaning you can inspect, modify, and redistribute the code freely.
To apply a pre-fitted lens, you load a HuggingFace model, wrap it with the jlens library, load a pre-computed lens, and run it on prompts of your choice:
import transformers, jlensmodel = jlens.from_hf(hf_model, tokenizer)lens = jlens.JacobianLens.from_pretrained('org/lens-repo')lens_logits, model_logits, _ = lens.apply(model, 'Your prompt here', positions=[-2])
To fit a lens on your own model: lens = jlens.fit(model, prompts=my_prompts). This is a reference implementation and is not optimized -- fitting time is dominated by the model's own backward pass. You can parallelize by running fit() on disjoint slices and combining with JacobianLens.merge(). A complete walkthrough notebook is included in the repository.
The tool works with any open-weights decoder transformer. The examples use Qwen, but the README notes that other HuggingFace decoders adapt cleanly with minimal changes.
How It Compares to Other Interpretability Tools
The interpretability toolkit landscape has grown quickly over the past two years. TransformerLens (by Neel Nanda and Joseph Bloom) offers cached activations and logit-lens-style probes for research. Logit Lens itself has been around since 2022, but only captures the final-layer decoding. The Jacobian Lens is distinct because it transports intermediate layer activations through the average Jacobian, not just the unembedding matrix. This gives higher-fidelity readings at middle layers where the Logit Lens tends to produce gibberish.
Anthropic's own earlier work on dictionary learning (feature visualization via sparse autoencoders) goes deeper -- it actually decomposes individual neurons into interpretable features. But sparse autoencoders are expensive to train and harder to deploy. The Jacobian Lens is lightweight by comparison: you can fit it on a single GPU in hours, not days.
OpenAI's work on activation patching (as seen in their GPT-2 interpretability work) is complementary but requires causal intervention in the model's forward pass, which is invasive. The Jacobian Lens is purely observational -- it reads activations without modifying them.
The key differentiator: the Jacobian Lens is the first tool that gives you layer-by-layer, position-by-position vocabulary-level readings with interactive visualization, all as a pip-installable open-source package with under 1,000 lines of core code.
Who This Is For
This tool serves three distinct audiences:
AI safety researchers get a practical probe for understanding how models represent knowledge internally. If you are studying truthfulness, sycophancy, or deception in language models, the Jacobian Lens gives you a window into whether the model 'knows' something before it says it.
ML engineers building on open-weight models get a debugging tool. When your fine-tuned model behaves unexpectedly, the Jacobian Lens can show you which layer the unexpected representation emerges from, helping you target your fine-tuning or prompting strategy more precisely.
Founders building high-stakes AI applications (healthcare, finance, legal) get a compliance and trust tool. Regulators will increasingly ask: 'How does your model arrive at its decisions?' Having a Jacobian Lens trace showing that the model internally represented the correct medical condition before outputting the wrong dosage is evidence of where the failure happened -- and where to fix it.
The repo is at github.com/anthropics/jacobian-lens under Apache 2.0. At the current growth rate (1,445 stars in 16 days), it will cross 3,000 stars within a month. If interpretability is the key to unlocking trust in AI systems, this release just made that trust significantly more accessible.