https://arxiv.org/abs/2509.26182

Why this matters

First things first: we want your personal AI agents to be sovereign. It should not upload everything it sees to a giant centralized cloud. What it learns about you should live as a portable local memory you control. That way you are not locked into a single model, and no one can access, alter, or shut it off without your permission. If you ever trust an assistant or companion with your life’s context, it should live with you.

We know local models are great for keeping your memory private while enabling fast iteration. But a single machine only goes so far. Hosting anything beyond about 8B is often not feasible on an everyday computer, and buying extra machines just to try a larger model is rarely sustainable. You end up settling.

What Parallax is

Parallax is the operating system for sovereign AI. Under the hood, it is a distributed runtime and service fabric that turns heterogeneous machines into one traceable service you can build on. When a model is too big for one host, it is sharded into contiguous layer slices and distributed across your laptop, a lab GPU, and a teammate’s workstation, all orchestrated as one service. Each request takes the fastest path on a single host, across a LAN, or over the public internet, without a public IP or matching hardware.

Parallax opens up a wide range of ways to host and run your own AI apps and agents that are completely your own, including coding copilots, personal assistants, vision and speech pipelines, and multi-agent simulations.

At launch, Parallax supports 40+ open models from 0.6B to trillion-class MoE on GPUs and Apple Silicon, across Windows, Linux, and macOS.

Key capabilities

Parallax is not just another local LLM runner. It is a path to open intelligence, scaling from one desk to clusters and out to the world.

3 modes, 1 operating system: LocalHost, Co-Host on LAN, Global Host over WAN. Start on your own machine for smaller models and join clusters when you need more headroom.
Heterogeneity by default: run 40+ models across GPUs and Apple Silicon on Windows, Linux, and macOS.
Consistent performance: sustained throughput with tight tails under real WAN variability and high concurrency.
Network-aware scheduling and routing: contiguous layer shards placed via dynamic programming with water-filling, then per-request DAG routing uses RTT profiling to choose the fastest path. The system is churn-aware and reallocates in milliseconds.
Traceability embedded: deterministic execution, isolated peer execution, and per-request routing traceability for auditability and attribution.

What Parallax enables

Parallax orchestrates heterogeneous machines into an adaptive mesh that finds the fastest path per request and reorganizes under load. It supports three modes, each with distinct optimizations.

Mode 1: LocalHost (single-host)

Run models on your own machine with data-center-class responsiveness.

Usage scenario: personal agents

Mode 2: Co-Host (multi-host cluster)

Join a cluster with other hosts on the same LAN or private L2 or L3 segment.

Usage scenario: trust-circle clusters within small teams or families

Mode 3: Global Host (WAN-scale fabric)

Form a wide-area cluster across unmanaged networks.

Usage scenario: service-grade LLM serving across the globe

Together, these modes form a substrate for all sovereign AI applications to keep data local, scale when needed, and stay verifiable at every step.

The Sovereign AI OS

Parallax is more than a way to serve models. It is a practical foundation for AI applications that can stay sovereign and open. By turning a mix of Macs and PCs into one adaptive service, it lets builders keep memory local, use open tools, and collaborate across devices without sending everything to a central cloud. The system is traceable end-to-end, so results are reproducible and auditable. With Lattica in place today, and verification and multi-agent layers coming next, you can build and run coding copilots, private-memory agents, retrieval over your own files, and vision or speech pipelines that move with you from a home setup to a lab cluster to a global fabric using the same code.

Popular dev tools and agent frameworks can point to your Parallax-hosted endpoints, so you can bring apps like vibe coders, personal assistants, and agent IDEs into your own environment while staying on consumer devices.

How the OS works

Real hosts and links are uneven. Parallax profiles device performance and links RTTs, then routes through the world as it is. Three pillars are making the experience possible.

1) Scheduling: placing shards & routing requests

Split an NP-hard problem into two quick ones. Mixed hardware and uneven links make round-robin a dead end. We decide shards and placement first, then pick the best path for each request.

Phase 1: Model allocation

Partition the model into contiguous layer slices and map them to hosts with a dynamic programming plus water-filling solver. Our objectives:

Shallow pipeline depth to reduce latency.
Enough replicas to raise throughput when needed.
Balanced stage runtimes so fast machines do not idle behind slow ones.

The result is a hardware-aware layout that respects VRAM and FLOPs and minimizes activation hops.

Figure 1: Example of the first phase model allocation among heterogeneous GPU types across different geographic regions.

Phase 2: Path selection

Maintain a layer-indexed DAG whose node weights are profiled per-layer latencies and whose edge weights are inter-node RTTs. For every incoming request, a DP router chooses the minimum-latency path and will select a different path for the next request if conditions change. It naturally routes around congestion and slow links. Then, requests are executed by streaming hidden states through the layers. A host runs its slice, forwards activations to the next, and so on until the pass completes.

Figure 2: Example of the second phase GPU pipeline chain selection among GPUs (pipeline stages).

Practical features:

Heterogeneous hardware awareness: device profiles for FLOPs, VRAM, and bandwidth prevent the straggler effect.
Churn-aware and self-healing: joins, leaves, or slowdowns trigger localized reallocation in milliseconds (≤ 10 ms at 256 nodes in simulation). Routes refresh quickly via the DHT. Overhead stays in the low single-digit ms range.
Proven at useful scales: We’ve tested 7-node real deployments with algorithmic scalability to 256 nodes, supporting 20+ nodes and hundreds of concurrent requests while keeping tails tight.

How it applies across modes:

Local host: Phase 1 within one device profile.
Co-Host (LAN): Phase 1 across nearby hosts with similar link characteristics.
Global (WAN-scale): Both phases active. Model shards span LAN clusters with WAN-aware routing.

2. Communication: peer-to-peer discovery without friction

Find the best path, even behind NAT.

LAN auto-detection: prefer low-RTT local paths automatically.
Lattica transport with NAT traversal: peers join without public IPs, working across common NATs and firewalls to form WAN-scale pipelines.
Unified Protobuf data plane: activations and KV shards move consistently across GPU, CPU, and Apple Silicon backends.
Membership and health via DHT: peer discovery, latency and health sharing, and self-healing under churn. These signals feed scheduling for fast reallocation and route refresh.

How it applies across modes:

Co-Host: Prioritize LAN paths.
Global: Stitch LAN islands over WAN via Lattica.

3. Backends: make every host pull its weight

Common features

Continuous batching to keep devices saturated under bursty traffic.
Dynamic KV management for high concurrency.
Sharded model loading so each host loads only its assigned layers.
Prefix cache (radix tree) to accelerate shared prompts.
On the horizon: runtime weight updates and speculative decoding.

Apple Silicon

MLX-LM integration with large, continuous batching.
Advanced KV block allocation, including sliding window attention, so Macs are efficient decoding nodes.

NVIDIA GPUs

SGLang integration via runtime patching for compiler and kernel scheduling benefits without model code changes.
CUDA Graphs in decode to reduce launch overhead and lower token-level latency, especially for long contexts.

Dreaming bigger with Parallax

We tested Parallax under mixed hardware with inter-node RTTs around 11 to 14 ms to benchmark its performance as the OS of sovereign AI, as well as a backbone for Open Intelligence.

Experiment setup: 14-node Global host (WAN-scale) run

Model: Qwen3-32B-FP8
Cluster: 14 NVIDIA RTX 5090 nodes, heterogeneous conditions
Throughput: ~495 tokens/s end to end; more than 1 req/s with ShareGPT-like traces at 200 concurrency
Latency: median inter-token 110 to 120 ms, p99 < 300 ms
Scaling efficiency: about 3× throughput vs a smaller non-optimized 4-node cluster on the same workload

Figure 3: End-to-end latency comparison between Parallax and HexGen across different models, traces, and request arrival rates.

Figure 4: End-to-end throughput comparison between Parallax and HexGen across different models, traces, and request arrival rates.

Figure 5: Phase-1 and phase-2 algorithm running time when scaling from smaller clusters (e.g., 4 GPUs) to larger clusters (e.g., 256 GPUs).

Head-to-head results summary (selected)

Comparison	Scenario	Metric	Parallax	Advantage
Global host vs HexGen	WildGPT/32B, rate 32	p99 latency	78.1 ms	2.6× lower
Global host vs HexGen	WildGPT/32B, rate 32	Throughput	0.40 req/s	3.6× higher
LAN Co-host vs Exo	Llama-3.1, 2048 in / 128 out	TTFT	4,532 ms	1.97× faster
Single GPU Local Host vs llama.cpp	RTX 5090, Qwen3-32B	TPOT (decode)	85.98 ms/token	1.41× faster

Parallax sustains high throughput with tight tails over WAN variability, not only on controlled LAN setups. In the runs above, it outperforms decentralized baselines and surpasses LAN-only prototypes, while keeping scheduling overhead under 10 ms at 256 GPUs and reallocating in milliseconds under churn.

That combination of contiguous sharding with DP and water-filling, per-request DAG routing with RTT profiling, and Lattica NAT traversal behaves like a service, not a demo. Hosts can join from anywhere, models can span them cleanly, and requests take the best path in real time.

Taken together, these properties make Parallax a hyper-scalable operating system for all sovereign AI applications.

What’s next

More features coming for Parallax:

Mixture-of-Experts aware scheduling: place experts and gates across hosts to keep hops low and throughput high while preserving sparsity.
Elastic sequence parallelism: scale context and tokens by flexibly splitting work across shard boundaries, adapting to runtime load.
Long context decode optimization: cut token latency at long lengths with improved KV handling and decode graph reuse.

Closing

Parallax isn’t just local AI. It’s an operating system built for multiplayer. Start on one box, co-host across a few machines, and scale to a global fabric so you can host all the AI applications you want and keep them sovereign.

Host your own AI cluster with 40+ models: github.com/GradientHQ/parallax
Read the technical paper: https://arxiv.org/abs/2509.26182

The age of scaling parameters is ending. The age of scaling sovereign AI starts here.