What We Are Building
A 3B Model That Thinks High Above Its Weight. That's just the starter.
First announcement of new architecture. We are building a 3.2 billion parameter language model called Sharona. Not inference training, building a new one. On paper, 3B is small -- the kind of model most teams use as a toy before scaling up. We are not scaling up. We are building an architecture that makes 3B punch like 13B on consumer hardware.
The architecture is multi-context. It splits the model into two processing paths with rolloff. One handles the live conversation. The other holds the additional knowledge and chat history -- compressed at roughly 19:1 using our deep semantic compression format. A 228K token compressed context holds the equivalent of 25.8 million natural language tokens. That is approximately 50 textbooks of grounded, searchable knowledge available to the model at inference time.
The model does not retrieve this knowledge through RAG or external database calls. It reads it directly through cross-attention during the forward pass. The knowledge is part of inference, not a bolted-on sidecar like those weird passenger motorcycles. It still supports RAG or Atlas.
What Makes This Different
Standard language models are generalists. They know a lot of things loosely. They have no mechanism to hold dense, domain-specific knowledge at inference time beyond what fits in their context window -- typically 128K to 200K tokens, uncompressed.
Sharona holds 25.8 million equivalent tokens at 3B. A 13B version would hold 58 million. A 70B version would hold 226 million. The knowledge scales with the model, and the hardware cost stays a fraction of what the equivalent standard model requires.
At full precision (FP16), the 3B model runs on consumer GPUs. A standard 13B model -- the performance equivalent -- needs hardware costing significantly more. At 4-bit quantization, Sharona 3B runs on modest hardware that most people already own.
The indexing is handled by QIPI, our custom indexing engine. QIPI achieves O(1) constant-time lookups -- not logarithmic like B-trees, not linear like brute-force vector search. On a 2017 desktop CPU, QIPI benchmarked at 708x faster than B-tree indexed lookups. Inside GPU memory, where the index lives alongside the model weights, a conservative estimate puts it at roughly 10,000x faster than PostgreSQL searching equivalent data from disk. At that speed, searching 25 million tokens adds less latency than a single attention layer.
QIPI is not a retrieval system in the traditional sense. It does not search for relevant context at query time. The knowledge is pre-loaded, pre-indexed, and wired directly into the model's attention. QIPI indexes it. The model attends to all of it, all the time.
The Open Source Plan
When we release the first model, it will ship with lite versions of our core technologies. The lite versions contain zero proprietary code -- they are built entirely on open-source foundations (SQLite, BM25, standard chunking). They still outperform most existing retrieval and grounding solutions. They just do not match what our proprietary engine does internally. The engine itself is for our future foundational model.
Atlas Lite -- the knowledge grounding layer, handling 3:1 chunking and compression for the public release. Genuinely useful, not a crippled demo.
QIPI-Lite -- a SQLite FTS5 BM25 retrieval engine. Deterministic, fast, zero external dependencies. Handles indexing and search for the compressed knowledge base.
Triple-context memory (M1/M2/M3) with QIPI-Lite indexing -- M1 is the hot conversation, M2 is warm compressed knowledge that persists across sessions, M3 is a recycle bin for decayed data before permanent deletion. Every model you use, including GPT, Claude, etc are M1 only.
The CE versions are designed to be useful on their own. Atlas CE and QIPI-Lite are real tools, not marketing wrappers. They run locally, require no cloud, and ship within the model.
The full internal stack -- including the complete QIPI engine with O(1) hash-based hybrid fusion, 19:1 compression, and the full Atlas retrieval pipeline -- stays proprietary. The public release gets the architecture. We keep the engine at foundational level.
The first model will be released on HuggingFace as Sharona 3B² -- B-squared, where the squared represents dual context multiplying capability. A 3 billion parameter model that performs at the level of a 13 billion parameter standard model. A fair comparison is Sharona 3B² against any other 3B model -- same parameter count, same hardware class. Comparing a 3B to a trillion-parameter model is not a benchmark, it is a category error. Within its weight class, the 3B² should outperform on both knowledge and reasoning -- not because the reasoning engine is smarter, but because it has 25.8 million tokens of grounded context to reason with. Same IQ, bigger library.
Power and Hardware
This is an efficiency play. The frontier labs are racing to make models smarter by making them bigger. We are racing to make intelligence and knowledge cheaper by building a more efficient architecture.
A Llama 3.3 70B FP16 requires 140GB of VRAM and $200K+ in datacenter hardware. It has access to 128K tokens of context.
Sharona 3B requires 4-8GB of VRAM and a consumer GPU. It has access to 25.8 million tokens of grounded knowledge. Sharona at this size will not compete on reasoning -- only a larger model can do that. However, on memory and knowledge grounding she will absolutely reign supreme. The math shows that in both reasoning and knowledge, our architecture at 400B² would rival a 2T model, and reign supreme on memory with capacity for 450M+ effective tokens. If you understand what that means, keep reading.
The 70B model draws hundreds of watts per inference. The 3B model draws what a gaming GPU draws during a loading screen. The knowledge capacity difference is not 2x or 5x -- it is 200x, running on hardware that costs 800x less.
Both models score well on standard benchmarks. But standard benchmarks do not measure knowledge capacity, grounding accuracy, retrieval speed, or cost-per-inference. They measure what the frontier labs are good at and ignore everything else. The 3B² we are currently building from scratch is the bottom rung to prove the architecture quickly. The bigger the model, the longer it takes to train. On our hardware, a 3B² takes about 8 days from scratch to finish.
That is why we proposed DCKG-Bench (Dense Contextual Knowledge Grounding) -- a benchmark that measures what matters for deployed, knowledge-intensive applications: how much knowledge can your model ground, how accurately, on what hardware, at what cost.
Where We Are Right Now
Training is in progress. V4 of the model is at step 3600 of 15,258, loss 3.68, perplexity 39.6. That is 23.6% through training. Each version comes with planned improvements rather than attempting to pile it in all at once. It is a meticulous and time-consuming process because every new addition presents possible failure points during training that we have to find ways to overcome. We can days into the build before seeing a fail point, causing a fix and restart.
The loss curve on V4 is healthy:
| Step | Loss |
|---|---|
| 100 | 7.54 |
| 500 | 6.25 |
| 1000 | 5.25 |
| 2000 | 4.50 |
| 2500 | 4.20 |
| 3100 | 3.97 |
| 3600 | 3.68 |
| 3800 | 3.53 |
| 3900 | 3.60 (slight bump, normal) |
No collapse. No divergence. Steady descent.
The chat output is still in what we call the babble phase -- similar to how infants acquire language. Early checkpoints produced random token soup. By step 1300, function words emerged (the, and, of, to, in). By step 2500, content words started appearing. By step 3100, we saw "a significant" placed correctly after a story prompt. By step 3600, "a great" and the discourse marker "When" appeared in contextually appropriate positions.
The model is learning word-class awareness -- understanding that certain slots in a sentence demand certain types of words. This is the hard part of a novel architecture. Standard models learn representation in one consistent space. Ours has to learn a harder mapping: think in compressed semantics, speak in natural language.
We are anxious to see if this works out with all final implementations.
Honestly, we have never built this architecture before, nobody has. The recipe we built -- dual-context with deep semantic compression, cross-decoder transplantation, custom indexing -- has never been done. When we asked AI assistants to help plan the training, the most capable ones said "I don't know" multiple times. That is not something you hear often from a model with a trillion parameters.
The ugly phase lasts longer than a standard model because there is more for the model to figure out internally. But the signals are there. The loss is healthy. The vocabulary is expanding. The word placement is improving. V1-V3 looked hopeless until step 4,000-8,000, then rapidly improved once the core mappings click. Of course, that could change since we continue adding in a little more on each build.
By 50% of training (~7,900 steps), we expect to still see the usual short, mostly coherent sentences on simple prompts. If the build is sound, that is where the babble clears. If not, we debug the transplant, the cross-attention, the data mixture, the hyperparameters, or recent additions. We have detailed logs at every checkpoint. The data is there to iterate.
If V4 completes as expected, we will build V5 as the final 3B open-source model and release it. The self-decoder is trained on 9 trillion tokens of web-scale data -- so the base knowledge is already world-class out of the box. On top of that, the model ships with lite versions of our core stack: Atlas CE for knowledge grounding and QIPI-Lite for retrieval. You may still want to fine-tune with pytorch for domain specialization, behavioral style, or task-specific performance. But thanks to our planned integration of tinygrad in V5 as the inference backend, the model should be fairly plug and play on any system -- NVIDIA, AMD, Apple Silicon, or even mobile GPUs.
What We Did Not Plan
We did not set out to build a model. We built SAIQL -- a fast, deterministic query engine. That was the product. Then compression emerged from the retrieval work, and that became the product. Then Atlas tied retrieval and grounding together, and that was the product. Then QIPI made it incredibly fast, and that was the product.
At each step, we thought we were done. At each step, the software told us what it needed next. Good architecture does that -- it reveals the next problem when you finish solving the current one. You cannot see over the next hill until you get there.
A database engine became a compression format. A compression format became a knowledge grounding layer. A grounding layer became a dual-context architecture. And a multi-context architecture, it turns out, is a new way to build a model.
Eighteen months of building has taught us to trust the pattern: solve the problem in front of you, build it well, don't skip or skimp, and the next problem reveals itself. We have never been able to see over the next hill. We are curious what it shows us this time. What comes after a new model architecture?
The Bet
The frontier labs are spending hundreds of millions of dollars to make models bigger. We spent $15,000 over two years -- including hardware, power, and all operating costs -- to build an architecture that makes models smaller while holding more knowledge.
Our 3B model on consumer GPUs will outperform a 13B model on significantly more expensive hardware in every knowledge-intensive task. A 13B² version would compete with 70B models that need $60,000+ in A100s. Not by being smarter per parameter, but by having 200x more grounded knowledge available during inference.
The frontier is racing to make intelligence bigger, more expensive. We are racing to make intelligence and knowledge cheaper in both hardware and power consumption by building a more efficient architecture.
Both matter. But only one of them lets you run on hardware you actually own. Can others do this? Probably. They'd have to come up with a deterministic memory system and some serious semantic compression like we have, but they could do it. Transplanting isn't new, neither is dual context. The tech we developed to make it worthwhile however, are very serious moats to overcome.
There is also a broader reason this matters. The AI industry is converging toward a handful of companies that control the models, the infrastructure, and the pricing. If that consolidation completes, every business that depends on AI becomes a tenant. We do not have a price. We are not building this to get acquired. We are building it because the world needs more than three companies deciding what AI costs and who gets access.
That is the bet. We are all-in. We are planning the next foundational. Here is our promise. With investor help. We are self funded right now. We will build rivals that do everything current foundation models do, but always remain at least 25% cheaper, and continue to offer open source models. The open source model runs locally and handles the majority of tasks on your own hardware. While the industry pours billions into the Apple IIe era of AI, we are building the 486 Turbo.
Want to see what Atlas already does today? Visit ShipItClean.com -- our AI security scanner holds entire codebases in memory, including the 44 million token Firefox repository, so the model never loses focus or runs out of context. That is the same grounding technology we are building directly into Sharona. That is actually a perfect example of our earlier statement "Good architecture reveals the next problem when you finish solving the current one." Because of the evolution, there is no way to have gotten here without building ShipItClean. There are no shortcuts.
Apollo Raines is the founder of SAIQL. Larry Arnold is cofounder and IEEE Invited Speaker on AI/ML in Future Networks. We are developer first.