September 8, 2026 · 4 min read

Introducing Quartz Micro Preview

The from-scratch MoE pretrain we first talked about on August 30 has a name.

Meet Quartz Micro Preview

Not a fine-tune of an existing checkpoint — randomly-initialized weights, a DeepSeek-style fine-grained mixture-of-experts, trained from raw token data on a single consumer GPU. “Micro” because it's deliberately small (~1B parameters, 394M active per token); “Preview” because this first run is a proof of concept for the approach, not the final word on how far it can go.

The training stack: FP16 mixed precision (autocast + GradScaler); 8-bit AdamW (bitsandbytes) to keep optimizer state small; Full gradient checkpointing; Gradient accumulation; Single NVIDIA GTX 1660 Ti, 6GB VRAM — a Windows desktop, not a cluster. Same machine as the August 30 report, unchanged the whole way through.

Where the run stands

sft step 64 / 1,317  (4.9% complete)
train loss: ~4.3968–5.367
throughput: ~59 tokens/sec

Pretraining finished — this is the instruction-tuning phase now: 14,051 examples from databricks-dolly-15k, 1,317 optimizer steps total. This number is live — reload this page and it'll reflect wherever the run actually is right now, same feed as the Research page.

It has already survived two full power/network outages and resumed cleanly from checkpoint both times — resiliency was a first-class design goal from the start, not an afterthought.

What's next: instruction tuning

Quartz Micro Preview is a base model right now — it has no instruction-following behavior of its own, because unlike Prism, Amethyst, and Copal (LoRA fine-tunes of an already-instruct-tuned base), it was never anything but raw pretrain. Once the run completes, we're running a full-parameter supervised fine-tune — not LoRA — on databricks-dolly-15k, a human-written instruction dataset. At ~1B total / 394M active parameters, a full fine-tune is cheap enough to run on the same GTX 1660 Ti that did the pretrain — and LoRA's light touch is the wrong tool for teaching chat format and instruction-following from a standing start.

Same rule as every other release: a held-out base-vs-tuned evaluation the moment training finishes, published honestly either way.

— The Vertex AGI team