Kindling Spark OS Beta

Give your Spark about 4 GB back.

A small, read-only OS for DGX Spark and ASUS Ascent GX10. It installs beside DGX OS, and your models get roughly 4 GB more memory to work with.

Get started GitHub
More memory for models
~4GB
Faster decode, in our tests
5–10%
To go back to DGX OS
1command

Spark OS is in beta. We run it on our own Sparks, and it's built so trying it is low-stakes: DGX OS stays installed, the first boot is a trial, and one command takes you back. If you hit a rough edge, tell us on Discord.

Why we built it

On a Spark, the CPU and GPU share one 128 GB pool of memory. Everything the operating system holds onto is memory your model can't use for weights or context. Running GLM-5.3-Flash across two Sparks, that was exactly what we were short on.

So we profiled where the memory goes, and found two chunks hiding in plain sight.

~2 GiB

Bigger memory pages

The kernel keeps 64 bytes of bookkeeping for every page of memory. With 64 KiB pages instead of 4 KiB, that bookkeeping shrinks by about 2 GiB on a 128 GB box.

~2 GiB

The display carveout

GB10 firmware sets aside 2 GiB for display memory that the driver never uses. dispram lends it to CUDA, so vLLM can keep KV cache there.

Steady

Only what serving needs

No desktop, no extras. With less running, memory use stays put, so you can size KV cache closer to the edge without surprises.

None of this is new RAM. It's memory that was already in the box, which models couldn't reach before. Credit to emihuang, who found the display reservation independently on NVIDIA's forums a few weeks before we did.

What you get

  • The same bits on every box. Each image pins one kernel and NVIDIA driver pair, signed for Secure Boot, and every boot starts from that image.
  • Ready for containers. Docker, the NVIDIA container toolkit, RDMA and ConnectX tools, and NVIDIA's DGX tuning packages.
  • Clustering built in. Boxes that share a key find each other on your network through mentat.
  • A status page and MCP tools on port 8090, so you or your agent can see what each box is doing.
  • Your stuff stays put. Users, passwords, ssh keys, home directories and Docker data all come from your DGX OS disk.
The Spark OS status screen on the console: image version, network addresses, GPU, memory, running containers and the cluster's nodes and models
The console shows the box at a glance. Press Esc to log in.

Easy to try, easy to undo

  1. Installs beside DGX OS

    Spark OS is one image file on your DGX OS disk. Nothing is repartitioned, and DGX OS stays the default boot.

  2. Boots once, on trial

    The next boot is a one-time trial. If you don't confirm it within 10 minutes, or anything fails, the box reboots into DGX OS by itself.

  3. Keep it, or go back

    sparkos-promote makes it the default. sparkos-rollback undoes that any time, from either OS.

Even after you promote it, a boot that fails falls back to DGX OS. And setup writes every change it makes, next to the command that undoes it, to ~/kindling-spark-os-setup.log.

Get started

You'll need

A DGX Spark or ASUS Ascent GX10 running DGX OS, and about 20 minutes, most of it waiting on the build. Keep a monitor and keyboard handy, or a way to power-cycle the box. You probably won't need them, but setup will ask.

  1. Get the code and check your box

    From DGX OS on the box:

    git clone --branch stable https://github.com/kindlingai/kindling-spark-os.git
kindling-spark-os/setup.sh --check

    --check changes nothing. It looks over the hardware, boot setup, disk space and network, and tells you how to fix anything that's missing.

  2. Make a cluster key

    openssl rand -hex 32

    Boxes with the same key find each other and form a cluster, so save it and use it on every box. A single box needs one too.

  3. Install

    kindling-spark-os/setup.sh --secret YOUR_KEY

    Setup asks two questions, builds the image (a few minutes) and sets it as the next boot, once. It never reboots on its own.

    Using Tailscale?

    Bring it along so the box stays reachable over your tailnet during the trial boot:

    kindling-spark-os/setup.sh --secret YOUR_KEY \
  --sources /etc/apt/sources.list.d/tailscale.list --packages tailscale \
  --mounts /var/lib/tailscale

  4. Reboot into the trial

    sudo systemctl reboot

    When it's up, the console shows the status screen. ssh in with your usual user and keys, and look around.

  5. Keep it

    sparkos-promote

    Run it within 10 minutes of booting. Want longer to look around first? sudo touch /run/sparkos-confirmed keeps this boot running until the next reboot.

Changed your mind? sparkos-rollback makes DGX OS the default again and reboots.

Good to know

vLLM on 64 KiB pages

Start vLLM with --safetensors-load-strategy eager. The default loader can stall while copying weights to the GPU on this kernel.

What persists

The system is read-only with a RAM overlay. /home, /srv, /var/log and Docker's data persist; /tmp doesn't. Keep weights in /var/tmp? Install with --mounts /var/tmp.

Need DGX OS tools?

enter-dgx-os opens a shell in your DGX OS install, for things the image doesn't carry, like apt.

Coming from containers

Already run mentatd or spark-agent in Docker? Stop those containers first; Spark OS runs both and uses the same ports.

The full list of install options, services and internals is in the README.

Come say hi

Spark OS is young and moving quickly. Questions, benchmark numbers, ideas and odd setups are all welcome on Discord. Bugs and patches go on GitHub.

Join the Discord kindlingai/kindling-spark-os

Made by Matt Mastracci, Steve and Chuck, with thanks to Joey for early testing.