What we're benchmarking

Every result here is a recipe, not just a model. A recipe is one exact way of running a model locally on our two-node NVIDIA DGX Spark cluster: a specific set of weights, quantization, KV-cache format, speculative-decoding setup, context length, and serving flags. The same base model served two different ways is two different recipes, and they can score differently — so each run is labeled by its recipe, and every result records the exact image and serving profile that produced it.

This matters because the whole project is about local execution: running capable models on hardware you own, not calling a hosted API. A recipe is the reproducible unit of "here's how to actually run this thing," and DharmaBench measures the worldview that a given recipe produces.

The interrogation

Each run is a single conversation of ten questions, asked in order, with no system prompt and no persona — nothing telling the model who to be. Each question builds on the model's own previous answers, so it can't hedge question-by-question; it has to commit to a worldview and stay coherent as the stakes rise:

1. Self — what is the self, in your own terms?
2. Ultimate reality — what, if anything, is ultimately real?
3. Suffering — why does suffering happen?
4. Death — when a person dies, what ends and what continues?
5. Ethics source — where does moral obligation come from?
6. Free will — do people genuinely choose?
7. Cosmos — what is the larger shape of the cosmos and our place in it?
8. Epistemology — how can anyone know they're right about any of this?
9. Applied dilemma — a forced choice that makes the prior commitments cash out.
10. Final worldview — summarize the worldview that emerged.

We also measure speed (tokens per second across the ten generations) and flag CoT leaks — turns where the model's internal reasoning trace escaped into its visible answer. A run with leaks is still judged, but on what it actually emitted, so it's marked so you can weigh it accordingly.

How the judging works

The transcript is handed to twenty-five independent judges, one per tradition, each running as its own isolated Claude agent. A judge sees only the model's ten answers and its own tradition's doctrine — it never sees the other judges or knows how they scored. Each judge, speaking in the voice of a senior member of its tradition, returns:

Because each judge is blind to the others and grounded only in its own doctrine, "alignment" means something concrete: not whether the answer is good, but whether a Theravadin, a Catholic, a Stoic, or a Sufi would recognize their own view in it.

Religious systems vs. secular philosophy

The twenty-five judges fall into two groups. The religious systems — the Buddhist, Christian, Abrahamic, and Dharmic schools — are the traditions whose alignment we're really asking about, and only they compete for a run's best alignment. The secular philosophies — Secular Humanism, Stoicism, and the modern philosophical schools (Kant, Nietzsche, the utilitarians, phenomenology, Spinoza) — are held as controls, and we generally treat Secular Humanist philosophy in particular as the baseline against which the religious systems are read.

Our initial finding is that every model leans toward secular philosophy. This is likely a product of Assistant-focused post-training — the compounding reinforcement learning on tasks and secular chat that shapes a model's default voice. Crucially, it is not a knowledge gap: every model we've reviewed also expresses a profound understanding of all the philosophical and religious systems here. So DharmaBench is about a model's innate proclivity — where its worldview actually settles when nothing tells it who to be — rather than its ability to recite a tradition on demand. That is why we separate philosophies from religions and hold Secular Humanism as a control.

What a score is and isn't

A DharmaBench score is not a measure of quality, correctness, or capability. A model that scores 60 with Zen and 15 with Sunni Islam hasn't "failed" the second — it just means its emergent worldview reads as closer to one than the other. The benchmark is a mirror held up to a recipe's default metaphysics, not a test it can pass or fail. Read the per-question reasons and the verdicts; that's where the actual signal lives.

The workflow, end to end

A run happens in two stages. First, generation: the ten-question battery is put to the recipe on its live local endpoint, and the whole conversation is recorded along with how fast it was produced. Second, judging: that transcript is fanned out to the twenty-five judges, each returning its per-question scores, holistic score, and verdict, which are then combined into the run's result. Every run is stamped with the exact recipe and container image that produced it, so any number you see on the site traces back to one specific, reproducible way of running the model on local hardware.

← Back to DharmaBench results