https://arxiv.org/abs/2511.11733

The Latency Wall

In centralized inference, speed is mostly a function of compute. You optimize by saturating HBM bandwidth, fusing kernels, and keeping GPUs close to their roofline.

In decentralized inference, where models are spread across home machines, university clusters, and diverse data centers, the physics change. The bottleneck shifts from compute to network latency.

When a model is sharded across the public internet, each decoding step becomes a distributed protocol. Let:

t0 be local compute time per token
t1 be point-to-point latency between nodes
N be the number of nodes
K be the number of generated tokens

With standard autoregressive decoding, producing every token requires synchronization across all nodes. The communication cost grows roughly as:

$$\begin{equation} T_{\text{comm}}^{\text{base}} \approx (N - 1) t_1 \end{equation}$$

If t1≫t0, GPUs spend much of their time idle, waiting for packets.

To build a scalable open intelligence stack on top of Parallax, we cannot treat latency as a fixed tax. We need to convert it into useful work.

This is the goal of Decentralized Speculative Decoding (DSD), which we have implemented inside the Parallax distributed inference engine. By verifying multiple tokens per synchronization and relaxing checks on low-impact tokens, DSD achieves up to 2.6x end-to-end acceleration over standard decentralized decoding, while maintaining accuracy on tasks like HumanEval and GSM8K.

DSD is built on two layers of innovation:

Batch Settlement: Amortizing communication by verifying multiple speculative tokens in one round.
Adaptive Verification: A training-free rule that adjusts verification strictness based on token-level semantic importance.

Innovation Layer 1: Batch Settlement (Logistics)

Classical speculative decoding pairs a lightweight draft model with a stronger target model. The draft proposes a window of tokens, the target verifies them in a single forward pass, and the system accepts the longest matching prefix. This increases throughput even in a single-machine setting.

In a decentralized setting, the same mechanism has an additional role: it reduces the number of cross-node synchronizations.

Instead of verifying one token per round trip, Parallax uses Batch Settlement Decoding:

Draft Phase: A lightweight draft model generates a speculative window of γ tokens locally.
Verification Phase: The target model, distributed across nodes, verifies the window in one forward pass and determines how many of the first tokens can be accepted.
Settlement Phase: The first k≤γ accepted tokens are committed in one synchronization round. All shards advance by k tokens. Any remaining tokens in the draft window are carried into the next speculative step.

In this regime, the communication term becomes: TcommDSD≈⌈kK⌉(N−1)t1

The network cost is effectively divided by the average accepted span k. When k is in the range of 3 to 4 tokens, the number of global synchronizations drops by a similar factor.

Engineering for Real Networks

Making this work on real networks and heterogeneous hardware requires additional systems work. Parallax implements three concrete mechanisms:

Progressive Batching: Early in a generation, Parallax uses smaller draft windows to reduce time-to-first-token. As the sequence stabilizes, the window size γ is increased to amortize communication over more tokens.
Paged KV Cache: The attention KV cache is split into independent pages, similar to virtual memory. This avoids monolithic allocations that would cause OOM errors on heterogeneous cards and allows KV pages to be swapped across nodes in smaller units.
Idempotent Commit: Each speculative window carries a DraftID and version. Commit operations are idempotent, so packet retries do not corrupt state and duplicated messages do not double-apply accepted tokens. This is important when Parallax spans regions and networks with non-trivial loss.

Together, these design choices turn batch settlement into a robust building block for decentralized inference, not just a theoretical optimization.

Innovation Layer 2: Adaptive Verification (Logic)

Batch settlement only delivers gains if the target model accepts enough tokens per round. Standard speculative decoding is conservative. It demands that the draft and target distributions match closely for every token in the window. A small mismatch anywhere truncates the accepted prefix.

Decentralized Speculative Decoding introduces Adaptive Speculative Verification. The key insight is that not every token has equal semantic impact.

Key Token Identification

For each position in the draft window, DSD computes:

The cross-entropy of the draft prediction.
The cross-entropy of the target prediction.
A normalized similarity score between the two predictive distributions.

A token is labeled as a Key Token if any of these signals indicates that the position is likely to influence future semantics. Typical examples include function names in code, numeric values in reasoning chains, and control symbols. Tokens that do not meet the criteria are treated as non-key tokens (e.g., filler words, punctuation).

Relaxed Checks for Non-Key Tokens

For Key Tokens: DSD uses the standard strict rule: the draft proposal must match the target distribution within a tight tolerance.
For Non-Key Tokens: DSD forms a softened version of the target distribution. A relaxation coefficient τ(x) scales how aggressively the distribution is flattened at that position.

Conceptually:

Semantically important tokens receive strict checks.
Low-impact tokens receive slightly relaxed checks that allow "good enough" matches.

This mechanism is training-free. It uses logits that are already available at inference time and does not require any finetuning. It is also model-agnostic and can be enabled for any draft–target pair that runs on Parallax.

In experiments, adaptive verification increases the average accepted span by roughly 0.8 to 1.2 tokens compared to non-adaptive speculative decoding, yielding an additional 15-20% end-to-end speedup at accuracy parity.

(Note: Adaptive verification is purely about inference dynamics. It is distinct from our VeriLLM layer, which focuses on cryptographic trust in the outputs.)

How DSD Performs in Parallax

We evaluate DSD inside Parallax using two open-weight models: Llama 3.1 8B and Qwen 3 8B, each deployed across 4 to 8 nodes with NVIDIA A800 80GB GPUs and 100 Gbps InfiniBand.

	alpaca		mt_bench		CNN/DM		Humaneval		GSM8k
	speed up	Avg acc len	speed up	Avg acc len	speed up	Avg acc len	speed up	Avg acc len	speed up	Avg acc len
Eagle3	2.108x	2.790	1.922x	2.706	1.643x	2.4509	2.313x	3.340	2.274x	3.403
DSD	2.345x	3.593	2.321x	3.432	1.905x	2.9871	2.562x	3.855	2.517x	4.042

We compare three strategies:

Baseline: Standard decentralized decoding without speculation.
Eagle3: Centralized speculative decoding adapted to our setting.
DSD (Parallax): With batch settlement and adaptive verification.

Cross-Task Results

Averaged across tasks (AlpacaEval, MT-Bench, CNN/DM, HumanEval, GSM8K):

Eagle3: Achieves ~1.6x - 2.3x speedup.
DSD: Improves this to roughly 1.9x - 2.56x, with higher average accepted length in every case.

On individual tasks:

HumanEval (Code): DSD reaches up to 2.56x acceleration with accuracy equal to or slightly higher than non-speculative decoding.
GSM8K (Math): DSD reaches up to 2.59x acceleration at accuracy parity.

From a systems point of view, DSD shifts the runtime budget: GPU utilization (MFU) increases, the share of time spent on communication decreases, and tail latency becomes tighter in high-RTT settings.

What This Means for Parallax Users

For users building on Parallax, Decentralized Speculative Decoding is exposed as an engine-level feature. Once enabled, you get:

Higher Throughput: More requests per second, or longer contexts, without increasing cluster size.
Lower Network Sensitivity: Fewer global synchronizations mean that WAN latency and jitter have less impact—critical when mixing data center GPUs with consumer devices.
No Changes to Your Models: DSD is training-free and model-agnostic.
Compatibility: DSD composes with our verification (VeriLLM) and routing layers.

In practical terms, agents, coding assistants, and research workloads running on Parallax will see snappier responses and better hardware utilization, especially when the underlying cluster spans multiple locations.