The Benchmark Nobody Asked For (That Changes Everything)
Every week, a new AI model climbs the leaderboard. AIME scores go up. SWE-bench results improve. GPQA-Diamond percentages tick higher. Press releases go out. The message is always the same: bigger is better.
But there is a question nobody on that leaderboard is answering: how much does it cost to actually use these models on real knowledge?
A 70B parameter model scoring 95% on a math olympiad is impressive. But it requires $200,000+ in datacenter GPUs to serve. A 600B parameter model with 128K context can hold a lot of text -- but try running it on hardware you can actually buy at Best Buy. The entire benchmark ecosystem rewards scale and punishes efficiency. If your model is small but architecturally clever, you are invisible.
We think that is backwards.
The Current Leaderboard Measures the Wrong Thing
Here is a snapshot of today's frontier model benchmarks:
| Benchmark | What It Measures | Min Hardware to Compete | Accessible? |
|---|---|---|---|
| AIME 2026 | Competition math reasoning | 8x H100 ($200K+) | No |
| SWE-bench Pro | Real-world code repair | 8x H100 ($200K+) | No |
| GPQA-Diamond | PhD-level QA | 8x H100 ($200K+) | No |
| HLE | Hard language evaluation | 8x H100 ($200K+) | No |
| DCKG-Bench | Knowledge grounding per dollar | GTX 1650 ($200) | Yes |
Every benchmark on that table except one requires datacenter-class hardware to even participate. They are measuring how smart a model is when you give it unlimited resources. Nobody is measuring how smart a model is when you give it a budget.
Introducing DCKG-Bench
The Dual-Context Knowledge Grounding Benchmark measures what actually matters for production AI systems: how much knowledge can your model access, how accurately can it reason over that knowledge, and what does the hardware cost to do it?
Instead of asking "can your model solve competition math," DCKG-Bench asks: "given 25 million tokens of real-world knowledge, can your model answer questions that require cross-referencing that knowledge -- and can it do it on a GPU that costs less than a used car?"
The Five Pillars
KDR -- Knowledge Density Ratio
How efficiently does your model compress and store knowledge? Raw text in a context window scores 1:1. Semantic compression architectures can score 10:1 or higher. More knowledge per token means more capability per dollar of hardware.
KCE -- KV Cache Efficiency
How much VRAM does your cached knowledge consume? Models that precompute and reuse knowledge representations amortize the cost across thousands of queries. Models that re-process everything per query pay full price every time. A precomputed KV cache turns a one-time cost into a permanent asset.
HCE -- Hardware Cost Efficiency
What is the cheapest GPU that can serve your model with its full knowledge base loaded? We define five hardware tiers, from a $200 consumer card to a $200,000+ multi-GPU cluster. Where your model lands determines who can actually use it.
CCGA -- Cross-Context Grounding Accuracy
The accuracy test. 500 questions across five difficulty tiers: direct retrieval, cross-referencing, synthesis, contradiction detection, and inference. This is where the model proves it can reason over its knowledge base, not just store it.
ENR -- Efficiency-Normalized Reasoning
The composite score that levels the playing field. A small model delivering 90% accuracy on a $200 GPU outscores a massive model delivering 95% accuracy on $200,000 of datacenter hardware. This is intentional.
Hardware Tiers
Every model evaluated on DCKG-Bench must declare the minimum hardware it requires to serve its full knowledge base. This determines its tier:
The Comparison Nobody Wants to See
Here is how different architectures would be evaluated on DCKG-Bench. All models are rated at full precision (FP16) -- no quantization. Quantized models can run on cheaper hardware at the cost of accuracy, but DCKG-Bench measures uncompromised performance. Values are estimated for illustration, but the structural differences are real.
| Architecture | Params | Eff. Context | Compression | Min VRAM | Tier | HW Cost |
|---|---|---|---|---|---|---|
| Dual-Context 3B | 3.2B | 25.8M | ~19:1 | 4-8 GB | Tier 1 | $200 |
| Llama 3.3 70B | 70B | 128K | 1:1 | 140 GB | Tier 5 | $200K+ |
| GPT-5.5 (API) | ~1T+ | 200K | 1:1 | N/A | Tier 5 | API cost |
| Claude Opus 4.8 (API) | ~500B+ | 200K | 1:1 | N/A | Tier 5 | API cost |
| Mistral 7B + RAG | 7B | ~50K eff. | ~3:1 | 8 GB | Tier 1 | $300 |
| DeepSeek-V4 (MoE) | ~600B | 128K | 1:1 | 300+ GB | Tier 5 | $300K+ |
Read that table again. The 3B model with semantic compression has access to 25.8 million tokens of grounded knowledge. The 70B model has access to 128 thousand. The 3B model runs on a GPU you can buy at Micro Center for $200. The 70B model requires a quarter-million dollars in datacenter hardware. And the benchmark everyone watches says the 70B model is "better."
The Test Suite
DCKG-Bench uses a standardized knowledge corpus of 25 million NL-equivalent tokens covering diverse domains: history, science, law, medicine, finance, literature. Each model must load this corpus into its available knowledge mechanism -- context window, RAG pipeline, KV cache, or compressed context.
Then it answers 500 questions across five difficulty tiers:
- Direct Retrieval (100 questions) -- Answer is a single fact in the corpus. Tests basic knowledge access.
- Cross-Reference (100 questions) -- Answer requires combining two facts from different sections. Tests multi-hop reasoning.
- Synthesis (100 questions) -- Answer requires reasoning across three or more facts. Tests deep knowledge integration.
- Contradiction Detection (100 questions) -- Identify conflicting claims in the corpus. Tests analytical reasoning over knowledge.
- Inference (100 questions) -- Answer requires implicit reasoning from corpus facts. Tests genuine understanding vs. pattern matching.
For each model, we measure: total VRAM consumed, time to load the knowledge corpus, average query latency, accuracy per tier, hallucination rate, and the exact hardware used. Every number is reproducible. Every result is auditable.
Why This Changes the Conversation
The AI industry has a scaling problem -- not a scaling-up problem, but a scaling-down problem. The most capable models in the world are locked behind API paywalls and datacenter infrastructure. They cannot run on edge devices. They cannot run in air-gapped environments. They cannot run where privacy, sovereignty, or cost constraints exist.
DCKG-Bench asks a simple question: what if the bottleneck is not model intelligence, but knowledge delivery?
A 3B parameter model with access to 25 million tokens of compressed, precomputed knowledge can answer the same domain questions as a 70B model with a 128K context window -- at a fraction of the cost, on hardware anyone can buy. The trick is not making the model bigger. The trick is making the knowledge smaller.
The models at the top of AIME and SWE-bench are extraordinary achievements in raw reasoning. But reasoning without affordable knowledge access is an academic exercise. DCKG-Bench measures the complete system: intelligence plus knowledge plus cost. Because in production, all three matter.
What We Are Building
We are developing a dual-context architecture that separates knowledge storage from language generation. Knowledge is compressed into a semantic format and precomputed into a KV cache that persists across queries. The language model cross-references this cached knowledge during generation, grounding every response in a knowledge base that would require millions of raw tokens to represent conventionally.
The result: a 3B parameter model on a single consumer GPU with the effective knowledge depth of models 10-20x its size. Not by being smarter at math. By being smarter about knowledge.
DCKG-Bench is not anti-scale. Large models that implement efficient knowledge grounding will score well on this benchmark. The benchmark is anti-waste. It penalizes architectures that require 100x the hardware to deliver marginally better answers. It rewards innovation that makes AI accessible to everyone, not just organizations with datacenter budgets.
The Benchmark Is Coming
We will publish the full DCKG-Bench specification, test corpus, and evaluation framework as an open benchmark. Any model can be evaluated. Any architecture can compete. The only question is: how much knowledge can you deliver, and what does it cost?
The leaderboard everyone watches measures intelligence.
The leaderboard that matters measures intelligence per dollar.
We are building the second one.
A Note on Honesty: 3B Is Tiny
We are not pretending a 3B parameter model matches a 70B model at reasoning. It does not. A 3B model is small. Its raw intelligence -- logical deduction, multi-step math, complex code generation -- is limited compared to frontier models with 20x or 200x the parameters. On standard benchmarks, it would get crushed. We know that. We are not hiding from it.
But that is not the point.
The 3B model is a proof of concept for an architecture, not a finished product. We started at 3B because it trains on two consumer GPUs in days instead of months. It lets us iterate on the dual-context design, debug the knowledge grounding pipeline, and validate that compressed semantic context actually works -- all without a datacenter budget. The architecture is what matters. The parameter count is what we could afford to test it on.
Where This Goes: The Scaling Ladder
The dual-context architecture with LoreToken compression is not locked to 3B. It is a design pattern that scales. As the base model grows, it brings stronger native reasoning -- and the compressed knowledge layer multiplies that reasoning across a vastly larger knowledge base than the model could otherwise access.
Our projected equivalence ladder, based on the compression ratio and knowledge grounding architecture. Select a precision to see hardware requirements:
| LoreToken Model | Eff. Context | Reasoning Equivalent | Min Hardware (FP16) | Std. Model Cost (FP16) |
|---|
The reasoning gap shrinks as the base model grows -- and so does the knowledge base. A 3B model has limited native intelligence, so the knowledge grounding does the heavy lifting across 25.8M tokens. A 13B model brings substantially stronger reasoning and handles a 58M token knowledge base to punch into 70B territory. A 70B LoreToken model with 226M+ tokens of grounded knowledge would compete with models that cost 100x more to serve -- because both the architecture advantage and the knowledge capacity compound with scale. At 400B, the LoreToken model would hold 450M+ tokens of compressed knowledge and reason at a level equivalent to a 2 trillion parameter standard model -- a model class that does not exist and cannot be served with any current infrastructure at any price.
Beyond the ladder: 2T + LoreTokens. At 2 trillion parameters, a LoreToken model would theoretically reason at a level equivalent to roughly 10-15 trillion standard parameters. That is speculative -- nobody has built either one, and the compression multiplier may not hold linearly at that scale. But the question itself is interesting: what happens when a model with very strong native reasoning also has compressed access to billions of NL-equivalent tokens of grounded knowledge? We do not know yet. We would like to find out. The architecture to start answering that question begins here, at 3B, on two gaming GPUs.
Projected Knowledge Capacity by Model Size
As the base model grows, so does its ability to handle larger context2 windows. Larger models have more attention heads, deeper representations, and better capacity to cross-reference across larger knowledge bases.
| Model Size | Max Context2 (LoreTokens) | NL-Equivalent | Knowledge Scale |
|---|---|---|---|
| 3B | 228K | 25.8M tokens | ~50 textbooks |
| 13B | 512K | 58M tokens | ~115 textbooks |
| 32B | 1M | 113M tokens | Small university library |
| 70B | 2M+ | 226M+ tokens | Full professional corpus |
| 400B | 4M+ | 450M+ tokens | Multiple professional domains |
"But Can You Actually Search That Fast?"
The obvious objection. A model carrying 25 million tokens of searchable knowledge sounds impressive until you ask how it finds the right information fast enough to not slow down inference. If the retrieval adds latency, the architecture falls apart.
The answer is QIPI -- Quantum-Inspired Probabilistic Index -- our custom indexer originally built for SAIQL. QIPI achieves O(1) constant-time lookups regardless of dataset size. Not logarithmic like B-tree indexes. Not linear like brute-force vector search. Constant. Whether the index holds 1,000 records or 25 million tokens of compressed knowledge, the lookup time is the same.
On a 2017 desktop CPU (i7-8700), QIPI already benchmarks at 708x faster than B-tree indexed lookups. For comparison, PostgreSQL -- the gold standard of relational databases -- manages about 16 operations per second on the same hardware. QIPI does nearly 10,000. That is the baseline, running on seven-year-old consumer hardware, outside the model entirely.
Inside the model, it gets dramatically faster.
Traditional search engines -- PostgreSQL, Elasticsearch, vector databases -- all operate through the CPU, hitting system memory or disk. Even the fastest NVMe SSD delivers about 7 GB/s of bandwidth. Standard DDR4 memory tops out around 50 GB/s. Every query crosses a bus, waits for a seek, marshals data between systems.
GPU high-bandwidth memory is a different universe. An RTX 3090 delivers 936 GB/s. An H100 delivers 3,350 GB/s. When the QIPI index lives in the same GPU memory as the model weights, there is no bus transfer, no disk seek, no network hop, no separate retrieval step. The lookup happens as a tensor operation during the forward pass. It is part of inference, not a sidecar bolted onto it.
This is the architectural point that matters: the knowledge and the index that searches it live in the same memory space as the model doing the reasoning. There is zero I/O overhead. A conservative estimate puts QIPI inside GPU memory at roughly 10,000x faster than PostgreSQL searching equivalent data from disk. At that speed, retrieving from 25 million tokens of grounded knowledge adds less latency than a single attention layer.
The knowledge is not baggage. It is native.
The 3B model we are building right now is the smallest rung on this ladder. It exists to prove the architecture works. Once proven, the same design transplants to 13B, 32B, 70B -- each time bringing stronger native reasoning and larger knowledge capacity, running on hardware that costs a fraction of what the equivalent standard model requires.
The frontier labs are racing to make models smarter. We are racing to make intelligence and knowledge cheaper in both hardware and power consumption by building a more efficient architecture. Both matter. But only one of them lets you run on hardware you actually own.
That is the bet.
How We Got Here
At every step of this project, we thought we had built the product. SAIQL was the product -- a fast, deterministic query engine. Then LoreTokens emerged from the compression work, and that was the product. Then Atlas tied retrieval and grounding together, and surely that was the product. QIPI was just the indexer we needed to make it all fast enough.
We never set out to build a new kind of AI model. Never considered it. We were building tools -- each one solving the problem directly in front of us. But good software has a way of telling you what it needs next, and if you listen, the pieces start fitting together in ways you did not plan. A query engine became a compression format. A compression format became a knowledge grounding layer. A grounding layer became a dual-context architecture. And a dual-context architecture, it turns out, is a new way to build a model.
We only see it in hindsight. At the time, each step felt like finishing. In reality, each step was foundation for something we could not see yet. We still cannot see over the next hill. We never have been able to. But eighteen months of building has taught us to trust the pattern: solve the problem in front of you, build it well, and the next problem will reveal itself.
That is how a database engine accidentally became a cognitive architecture. And honestly, we are curious what it tells us to build next.