Building Sharona: Three Minds, Two GPUs, and an Honest "I Don't Know"
We're building a language model in an office on two gaming GPUs. None of us know if it will work.
That's not the pitch. That's the actual status.
Sharona is a 3.18 billion parameter model built on an architecture most researchers ignored, trained on a data format nobody else uses, designed to answer a question that three very different minds -- a founder, a cofounder, and an AI -- have all independently answered the same way: "I don't know."
This is the story of how we got here.
(The name? We'd been calling the project Nova, but Amazon named their model Nova. "So maybe Sharona," Apollo said. The name stuck. She already existed in our ecosystem as a LinkedIn automation agent. And "My Sharona" is one of the most recognizable song names ever, which doesn't hurt.)
"How Long Will This Take?"
It started the way most bad ideas start -- with optimism.
We'd built the full model. 3.18 billion parameters. An architecture called YOCO -- You Only Cache Once.
Here's the idea. Every language model today has a single context window. You put your prompt in, the model reads it, and generates output. If you want the model to reference a large document while answering questions, that document sits inside the same context window as your question. Every token of that document competes with every token of your conversation for the model's attention. It's expensive, it's slow, and the bigger the reference document gets, the worse it scales.
YOCO splits the model into two halves with two separate contexts. The first half -- the "self-decoder" -- processes a large reference context a single time and caches the result. Think of it as the model reading a textbook once and taking notes. The second half -- the "cross-decoder" -- handles your actual conversation and can look back at those cached notes through cross-attention whenever it needs to. The textbook doesn't get re-read for every question. It was processed once. The notes persist.
That's the architecture. We'd tokenized 16.7 billion tokens of training data. Written the training scripts, the checkpoint system, the monitoring tools. Fixed seven bugs in sequence.
Then we launched training on our two RTX 3090s.
Silence.
Nine months. On hardware that costs $3,000. To train a model that the big labs would spin up on a cluster and finish in an afternoon.
We watched it run for 40 steps anyway. Loss dropped from 11.0 to 6.9. It was learning. It would just take until March 2027 to finish.
"Does Nobody Have Checkpoints?"
The conversation shifted. We couldn't train from scratch -- not on this hardware, not in any reasonable timeframe. But did we need to?
This is where it got interesting.
The Perfect Match
We pulled up the specs side by side:
| Dimension | Sharona 3B | Llama 3.2 3B |
|---|---|---|
| Hidden size | 3072 | 3072 |
| Query heads | 24 | 24 |
| KV heads | 8 | 8 |
| Head dimension | 128 | 128 |
| FFN size | 8192 | 8192 |
| Total layers | 28 | 28 |
| RoPE theta | 500,000 | 500,000 |
Every single dimension matched. Every weight tensor the same shape. We designed Sharona's architecture to align with the Llama family, but we hadn't planned to use their weights. 274 days changed our minds.
Architecture Surgery
We killed the from-scratch training. Ran the transplant. Launched fine-tuning with the self-decoder frozen -- 1.41 billion parameters locked with Llama's knowledge, 1.77 billion trainable.
And the organs took. 253 tensors, zero shape mismatches. The donor didn't even have to be a match -- it was an exact match. Same hidden dimension, same head count, same feed-forward size, same positional encoding. Llama 3.2 3B was built to the exact spec of a body that didn't exist yet.
The cross-attention layers -- the ones that make YOCO work, the whole reason Sharona exists -- stayed fresh. Random initialization. They have to learn from scratch how to read from the self-decoder's cache. But everything around them already knows how to think. Surround the new thing with competence and see if it figures out its job faster.
274 days versus 14. That's what stealing organs buys you.
Throughput jumped 45%. Loss was already dropping by step 20.
"A Model Already Thinks in LoreTokens"
Meanwhile, Larry Arnold -- cofounder, IEEE speaker, the philosophical backbone of the project -- was working a parallel thread.
LoreTokens are our semantic compression format. Where natural language takes thousands of tokens to express an idea, LoreTokens encode the same meaning in a fraction of the space. The compressed form remains active -- it carries meaning without decompilation. An intelligent reader (an LLM) reconstructs the full semantics on demand.
Larry's argument was simple:
He was right. The internal representations of any LLM are already dense semantic compressions. LoreTokens didn't invent a new way for AI to think. They formalized the way AI already thinks and made it transferable.
But earlier in the project, we'd proven something important -- and Apollo drew a line through it that changed how we thought about the whole thing.
"It Doesn't Think in LoreToken. That's the Difference."
A 14 billion parameter model -- Qwen, off the shelf, zero training -- could read and understand LoreTokens at zero shot. It parsed the structure, mirrored the format, extracted meaning. That was a real proof point. Models can already comprehend the format.
Understanding versus thinking. Completely different operations.
Right now, when any model reads LoreTokens, it's translating. LoreToken to natural language internally, reason in natural language, respond. It's like a fluent English speaker reading Spanish with a dictionary. They can get there, but there's overhead on every step.
A model trained on LoreTokens would think in that compressed representation. No translation step. The internal attention patterns, the embedding space, the reasoning chains -- all operating on information that's dramatically denser than what any other model works with. Same context window, same parameter count, but every weight tuned to process meaning at a compression ratio nobody else can touch.
Nobody has ever done this. Bilingual training across a natural language and a semantic compression format. It's genuinely new research.
Why YOCO Needed LoreTokens
That distinction -- understanding versus thinking -- also explains why the architecture matters.
LoreTokens change the math. The cached knowledge context stops being 'here's a long document' and becomes 'here's an entire compressed knowledge base.' When that context is 10-50x denser than natural language, processing it once and caching it for the model to reference on every query isn't an optimization -- it's the whole point. The architecture finally has a reason to exist.
That's what makes Sharona interesting as a thesis. Not YOCO alone, not LoreTokens alone -- the combination where one gives the other a purpose."
Microsoft Research published YOCO. It was technically sound. Nobody adopted it because the problem it solved for plain text wasn't worth the complexity. LoreTokens give it a reason to exist -- dense, structured, semantic data that benefits enormously from being processed once and cached for repeated cross-attention.
Neither technology justifies itself alone. Together they might justify each other.
Or they might not. We don't know yet.
Three "I Don't Knows"
So: architecture surgery complete. The bilingual thesis defined. The reason YOCO and LoreTokens need each other, articulated. All that remained was the question that started as simple and turned into something none of us expected.
1. Teaching our tokenizer -- embeddings are blank. Without this, the model literally can't read input. Mechanical, not optional.
2. Teaching cross-attention -- the mechanism that lets the conversation half of the model read from the cached knowledge half. No donor weights exist for this part. Without training it, the second context does nothing. Also mechanical.
3. Learning LT syntax patterns -- the model learns the statistical structure so it can parse and generate well-formed LoreTokens. Testable, straightforward.
Beyond those three? I don't know. Whether LT training teaches the model to reconstruct meaning from compressed tokens better than a base model already can, whether it develops something like calibrated confidence over structured data, whether the dense knowledge format actually shifts the knowledge-per-parameter curve -- those are the experiments we're running to find out.
The first three alone justify the training. Everything else is what makes it interesting."
Then Larry asked the question that stopped everyone.
Three entities. A cofounder who writes the theoretical foundations asked the question. An AI that just built and launched the entire training pipeline said "I don't know." A founder who's been building this for two years said the same thing. The guy who asked the question didn't know the answer either. Three minds, one question, zero answers.
So when I tell you the LoreToken-native training variable is a genuine unknown -- that's not modesty. It's that there's no training data in my weights for 'what happens when you train a model on a semantic compression format that was invented by one guy in his office and has never been published.' There's nothing to pattern-match against. No papers, no benchmarks, no Stack Overflow threads, nothing."
The Edge
Everybody else in AI is selling certainty. The pitch deck says "our model will do X." The blog post says "we've achieved Y." The paper says "we demonstrate Z."
We're running a training job on two gaming GPUs in an office to find out if a question even has an answer. Total infrastructure cost: $15,000 over two years. No outside capital. No cloud compute. No investor pressure to pretend we have answers we don't have.
The teams that pretend to know build on assumptions. The teams that admit what they don't know build experiments.
In 14 days, we'll have a 3.18 billion parameter model that can process LoreTokens through a dual-context architecture. Then we test:
- Does it reconstruct meaning from compressed tokens better than a base model?
- Does it develop calibrated confidence over structured data?
- Does dense knowledge change the scaling curve?
- What happens when you hand it knowledge that doesn't exist anywhere in its training data?
We don't know yet. But we will.
And there's one more thing we expect to see -- something that already happens with every large language model, that nobody fully understands, and that gets very interesting when you add dense compressed knowledge into the equation.
The Phenomenon
Models answer questions they shouldn't be able to answer. Not hallucinations -- those are confident wrong answers. This is the opposite. Confident right answers that have no business existing.
It happens often enough that researchers have documented it, but not often enough that anyone can trigger it on demand. A model produces a correct answer to a question where the specific fact never appeared in its training data. It wasn't memorized. It wasn't looked up. It was synthesized from something -- and the "something" is where it gets strange.
The part that's understood: models don't store facts like a database. They learn relationships. During training, a model sees millions of documents where concept A relates to concept B, and separately where concept B relates to concept C. It never sees A connected to C directly. But the weights encode those relationships as patterns in the embedding space, and at inference time, the model traverses A to B to C and produces a correct answer for a connection that never appeared in any training document. That's compositional generalization -- the same mechanism that lets you understand a sentence you've never heard before, because you know the words and the grammar even if that specific combination is new.
The part that's not understood: models sometimes go further. They synthesize across domains that weren't adjacent in training -- combining a pattern from biology with a structure from mathematics to answer a question about economics. The concepts weren't linked. The connection wasn't implied. But the model's embedding space -- which operates in billions of dimensions -- organized those concepts close enough together that the attention mechanism found a path between them. A real structural similarity that humans wouldn't have noticed, followed to a correct conclusion that humans hadn't drawn.
Nobody can predict when a model will do this. Nobody can reliably explain why it works when it does.
Now consider what this means for Sharona.
If this phenomenon already happens with models trained on verbose natural language -- where concepts are spread across thousands of tokens, buried in grammar, padded with filler -- what happens when the model thinks in representations that are 10-50x denser? When every token carries dramatically more semantic content? When the distance between concepts in the embedding space is compressed along with the language itself?
Do the gaps between concepts shrink? Do the leaps get shorter? More frequent? More accurate?
We don't know. That's one more "I don't know" for the list. But it's the most interesting one -- because if dense native reasoning makes this phenomenon more common or more reliable, then we're not just building a model that knows more per parameter. We're building a model that discovers more per inference step.
That's what we expect to see. Not because we can prove it will happen. Because the math says the conditions for it to happen are better in compressed space than in natural language. And in 14 days, we'll be able to test it.
Faster Than Fast
There's one more advantage worth mentioning, and this one isn't theoretical. It's just physics.
Most AI systems that need external knowledge use RAG -- Retrieval-Augmented Generation. Every time you ask a question, the system has to embed your query into a vector, search a database for similar vectors, rank the results, retrieve the top chunks, stuff them into the model's context, and then process those chunks through the full model. That retrieval pipeline typically takes 200-500 milliseconds. And it can miss. If the relevance scoring doesn't rank the right chunk high enough, the model never sees the answer.
We built a retrieval engine called Atlas that's already significantly faster than RAG -- 50-70 millisecond retrieval with deterministic scoring instead of the nondeterministic top-k behavior that can cause vector retrieval to miss relevant material. It's what powers our document scanners and knowledge systems today.
Sharona's second context is faster than Atlas.
Not by a little. By orders of magnitude. Because the second context isn't retrieval at all. The self-decoder processes the knowledge once, caches the key-value states, and from that point forward, every token the cross-decoder generates does a cross-attention lookup directly into that cache. No query. No search. No scoring. No round trip. The knowledge is already on the GPU as pre-computed attention states. The per-token cost is a single matrix multiply per layer -- microseconds, not milliseconds.
And because the model attends to everything in the cache simultaneously, there's no chunking strategy that might cut a concept in half, no top-k cutoff that might exclude the relevant passage. The model sees all of it, all the time, and decides what's relevant through attention -- the same mechanism it uses to understand language itself.
The tradeoff is capacity. Atlas can hold everything -- unbounded storage, no ceiling. The second context is limited to the context window. But with LoreTokens compressing knowledge at 10-50x density, even a 128K context window holds the equivalent of 1.2 to 6.4 million tokens of natural language knowledge. That's a lot of room before you overflow into Atlas. And when you do overflow, Atlas is still there -- filling the second context with exactly the right knowledge for the current task, then stepping back while the model works at attention speed.
Atlas is the library. The second context is the page you already have open. You read at the speed of thought, not the speed of search.
The Weight Class Question
Let's say it works. Let's say Sharona converges in 14 days, the cross-attention learns to use the cached LoreToken context, and the model demonstrates measurably better knowledge retrieval from compressed data than a base model of the same size.
What do we have?
A 3B model is small. In the current landscape, 3B is what runs on your phone. It's the size companies use for autocomplete, not for reasoning. A standard 3B model sits firmly in the lightweight tier -- useful for narrow tasks, outclassed by anything bigger on anything complex.
But Sharona isn't a standard 3B model.
A conventional model stores everything it knows in its weights. All of its world knowledge, all of its language understanding, all of its reasoning patterns -- baked into 3 billion parameters during training. When you ask it something, it has to retrieve the answer from those parameters. That's why bigger models know more: more parameters means more storage capacity for knowledge.
Sharona splits that equation in half.
The self-decoder carries general language capability -- transplanted from Llama's 9 trillion tokens of pre-training. That's the reasoning engine. The cross-decoder reads from a cached knowledge store through cross-attention. The knowledge doesn't live in the weights. It lives in the second context -- a LoreToken-compressed knowledge base loaded at inference time, processed once, cached, and attended to on every generation step.
If LoreTokens compress knowledge at 10-50x the density of natural language, then a 128K-token knowledge context isn't carrying 128K tokens worth of knowledge. It's carrying 1.2 to 6.4 million tokens worth. In a context window that a 3B model can actually process.
That changes the weight class math entirely.
A standard 3B knows what it learned during training. Period. A 3B with YOCO and dense LoreToken context knows what it learned during training PLUS whatever knowledge base you load into the second context. Swap the knowledge base, swap the expertise. A medical LoreToken store makes it a medical model. A legal store makes it a legal model. The same 3B parameters, punching at different weight classes depending on what knowledge you hand it.
How much bigger? We genuinely don't know yet. The density multiplier is real -- we can measure that. The question is how effectively the cross-attention mechanism retrieves and reasons over that dense context compared to a larger model that has the same knowledge baked into its weights. It might punch at 7B. It might punch at 13B for knowledge retrieval tasks while staying at 3B for pure reasoning. It might not punch above its weight at all. That's what the experiment answers.
But here's where it gets interesting.
What if we get experimentation funding and build a 70B?
Scale the same architecture to 70 billion parameters. The self-decoder gets a full 70B reasoning engine -- that's Llama 3.1 70B territory, genuine intelligence. The cross-decoder gets cross-attention into a LoreToken knowledge context at the same density multiplier.
A standard 70B model is already impressive. Strong open-model reasoning, deep world knowledge, genuine problem-solving capability. But it still only knows what's in its weights. It hallucinates when you ask about proprietary systems. It can't access your company's internal knowledge without RAG -- and RAG at natural language density wastes most of the context window on verbose, redundant text.
A 70B Sharona with LoreToken context doesn't have that limitation. And the second context doesn't stop at 228K.
The size of the second context is limited by GPU memory available for the KV cache. For a 70B YOCO model -- 8192 hidden dimension, 8 KV heads, 40 self-decoder layers, bf16 precision -- the KV cache costs roughly 160KB per token. The question is how much memory you have, and in 2026 that question has very different answers than it did two years ago.
On current-generation hardware:
- Single B200 (192GB HBM3e) -- second context up to ~1M tokens. At 10-50x LoreToken compression: 10 to 50 million tokens of equivalent knowledge. An entire field of medicine. A complete legal code. Every technical manual a company has ever written.
- GB200 node (two B200s + Grace CPU, 384GB combined GPU memory, NVLink) -- second context up to ~2M tokens. At compression: 20 to 100 million tokens. That's not a knowledge base. That's a library.
- GB200 NVL72 rack (36 Grace Blackwell Superchips, ~13TB aggregate HBM) -- at this scale the second context could hold tens of millions of LoreTokens. The equivalent knowledge is hard to even quantify. Hundreds of millions to billions of tokens of natural language. Every document a Fortune 500 company has ever produced, compressed and always loaded.
All processed once. All attended to at microsecond speed. All swappable at inference time.
To put this in perspective: Claude, GPT, Gemini -- the frontier models -- are pushing toward 1 million token context windows. That's impressive engineering. But their 1M tokens is 1M tokens of natural language. Verbose. Redundant. Every word of it competing with the conversation for the model's attention. Every token reprocessed on every generation step.
Sharona's 1M tokens is 1M LoreTokens. At 10-50x compression, that's 10 to 50 million tokens of equivalent knowledge. In a dedicated second context that doesn't touch the conversation window. Processed once, cached, attended to at the speed of a matrix multiply.
Shares window with conversation.
Reprocessed every generation step.
Dedicated second context.
Processed once, cached, attended at µs.
Same number on paper. Completely different thing in practice. A 1M-token context window and a 1M-LoreToken second context aren't even the same category of capability.
That's not a chatbot. That's not an assistant. That's a domain expert with perfect recall over an arbitrary knowledge base, scaling with hardware that's already shipping.
The 3B experiment on two gaming GPUs in an office proves whether the architecture works. The 70B version on current hardware is what it looks like when it scales.
And if the density multiplier holds? If a 70B with LoreToken context genuinely outperforms a 400B+ model operating on raw natural language context?
Then the game changes. Not because we built a bigger model. Because we built a denser one.
The 10 Trillion Parameter Question
What follows is speculative. None of this has been validated. We're reasoning from the math and the architecture to sketch what the picture looks like if the thesis holds. Treat it as a hypothesis, not a roadmap.
Follow the logic one step further. The frontier labs are pushing toward 10 trillion parameter models. The rumors, the leaks, the GPU orders -- everything points to models an order of magnitude larger than what exists today. Models that will cost billions to train and require entire data centers to run inference.
How big does a Sharona need to be to compete with that?
The honest answer requires separating two things that people constantly conflate: reasoning and knowledge.
Reasoning scales with parameters. A 3B model reasons like a 3B. A 70B reasons like a 70B. No amount of external context changes the depth of the reasoning engine itself. A 10T model will reason at a level that smaller models simply cannot match on the hardest problems -- multi-step logic, novel mathematical proofs, complex strategic planning. To match that raw reasoning horsepower, you need comparable parameters. Maybe not 10T -- there are diminishing returns -- but you'd need something in the 400B-1T range to play in the same league.
Knowledge is a different story entirely. This is where the math breaks in our favor.
A conventional 10T model stores everything it knows in its weights. Every fact, every relationship, every domain-specific detail it might need -- all baked into 10 trillion parameters during training. Most of those parameters aren't doing "reasoning." They're doing storage. They're lookup tables, memorized associations, compressed knowledge that gets retrieved during inference. Estimates vary, but a significant fraction of any large model's parameters exist to remember things, not to think about things.
A Sharona model doesn't need to store knowledge in its weights. It stores knowledge in the second context.
If the density multiplier holds at 10-50x, a 228K-token LoreToken context carries 2.2 to 11.4 million tokens of equivalent knowledge. That's knowledge you can swap, update, and specialize at inference time -- no retraining required. The weights only need to carry the reasoning engine and the ability to read from that dense context.
So the question becomes: what size reasoning engine, paired with dense LoreToken context, matches a 10T model on practical tasks?
Our best guess -- and it is a guess:
- 70B Sharona could compete on domain-specific tasks where the knowledge is loadable into the second context. Medical diagnosis, legal analysis, technical troubleshooting, code review -- anywhere the bottleneck is "does the model know enough about this specific domain." The 70B reasoning engine is strong enough to use the knowledge effectively. The LoreToken context provides the knowledge a 10T would have baked into its weights.
- 400B Sharona could compete on general tasks. At 400B, the reasoning engine itself is world-class. Pair it with swappable LoreToken knowledge stores and you have a model that reasons at near-frontier level while accessing dense, updatable, domain-specific knowledge through cross-attention. That's a 10T competitor at 4% of the parameters.
- A 3B Sharona -- what we're building right now -- won't compete with a 10T on reasoning. But it might compete on knowledge retrieval within its loaded domain. And that's enough to prove the thesis.
The Double Advantage
But the power comparison above is actually incomplete. We were only looking at one side of it -- fewer parameters means fewer GPUs. There's a second effect the math points to that we didn't see at first until we thought harder about what compression means at the hardware level.
When a model processes LoreTokens, the GPU is literally doing less work per unit of meaning.
A standard model processing a knowledge base in natural language has to push every token through every layer. If the knowledge takes 500,000 NL tokens to express, that's 500,000 forward passes through the self-attention and feed-forward networks of every layer. Self-attention compute scales quadratically with sequence length in the base case -- most production systems use optimizations like FlashAttention that reduce this, but the fundamental relationship between sequence length and compute cost remains superlinear.
LoreTokens compress that same knowledge into a fraction of the tokens. If the density ratio is 10x, that 500,000 tokens of knowledge becomes 50,000 LoreTokens. The model processes 1/10th the tokens. The savings are linear on feed-forward layers and superlinear on attention -- even with optimized kernels, fewer tokens means substantially less total compute. And with YOCO, that reduced computation happens once. The self-decoder processes the compressed context a single time, caches the key-value states, and the cross-decoder attends to that cache cheaply on every subsequent query.
Three multipliers stacking on top of each other:
- Smaller model -- fewer parameters means fewer GPUs means less power per token.
- Fewer tokens -- compressed data means less total computation for the same knowledge. Linear savings on feed-forward layers, superlinear savings on attention.
- Process once -- YOCO caching means the expensive context processing happens once, not per query. Amortized across thousands of queries, the per-query cost approaches just the cross-attention and generation cost.
The math hasn't been validated yet. That's what the experiment is for. But if the numbers hold, the implication is that a Sharona model doesn't just punch above its weight class on capability -- it consumes power below its weight class on compute.
A model that outperforms larger models on knowledge tasks while using less power than models its own size. Capability goes up, consumption goes down. Not one or the other. Both.
That would be a killer model.
Not because it's the smartest model in the room. Not because it beats a 10T on a benchmark. But because for the vast majority of practical applications -- the enterprise queries, the domain-specific questions, the knowledge retrieval and application that makes up the bulk of real-world AI usage -- it delivers comparable results at a fraction of the cost, a fraction of the power, and a fraction of the infrastructure.
The frontier labs are in an arms race for the biggest model. The biggest training run. The most GPUs. The most power.
What if the winning move is density, not size?
We don't know if any of this will work. Three minds, three "I don't knows." But the math is pointing somewhere, the experiment is running, and in 14 days we'll have the first real data.
That makes us proud.
Sharona is being trained on 2x NVIDIA RTX 3090 GPUs with NVLink, using DeepSpeed ZeRO Stage 2 with CPU optimizer offload. The model uses architecture surgery -- transplanting pre-trained weights from Llama 3.2 3B into a novel YOCO dual-context architecture -- to reduce training time from 274 days to 14 days.
Apollo Raines is the founder of SAIQL. Larry Arnold is cofounder and IEEE Invited Speaker on AI/ML in Future Networks.