September 8, 2026 · 4 min read
Introducing Quartz Micro Preview
The from-scratch MoE pretrain we first talked about on August 30 has a name.
Meet Quartz Micro Preview
Not a fine-tune of an existing checkpoint — randomly-initialized weights, a DeepSeek-style fine-grained mixture-of-experts, trained from raw token data on a single consumer GPU. “Micro” because it's deliberately small (~1B parameters, 394M active per token); “Preview” because this first run is a proof of concept for the approach, not the final word on how far it can go.
Total parameters
1,031.0M
Active parameters / token
394.0M
Hidden size
1,024
Layers
20
Attention
16Q / 4KV
Context length
2,048
Routed experts
24
Shared experts
2
Expert FFN size
640
Router
top-6 of 24
Vocabulary
32,000
The training stack: FP16 mixed precision (autocast + GradScaler); 8-bit AdamW (bitsandbytes) to keep optimizer state small; Full gradient checkpointing; Gradient accumulation; Single NVIDIA GTX 1660 Ti, 6GB VRAM — a Windows desktop, not a cluster. Same machine as the August 30 report, unchanged the whole way through.
Where the run stands
100,003,832 tokens of packed training data, 24,414 optimizer steps total. The run has finished or is between checkpoints right now — see the Research page for live status.
It has already survived two full power/network outages and resumed cleanly from checkpoint both times — resiliency was a first-class design goal from the start, not an afterthought.
What's next: instruction tuning
Quartz Micro Preview is a base model right now — it has no instruction-following behavior of its own, because unlike Prism, Amethyst, and Copal (LoRA fine-tunes of an already-instruct-tuned base), it was never anything but raw pretrain. Once the run completes, we're running a full-parameter supervised fine-tune — not LoRA — on databricks-dolly-15k, a human-written instruction dataset. At ~1B total / 394M active parameters, a full fine-tune is cheap enough to run on the same GTX 1660 Ti that did the pretrain — and LoRA's light touch is the wrong tool for teaching chat format and instruction-following from a standing start.
Same rule as every other release: a held-out base-vs-tuned evaluation the moment training finishes, published honestly either way.
— The Vertex AGI team