DoubleZero, Multicast Fiber

The bottleneck is jitter, not latency

Two different things get called "latency."

Latency is the median time a packet takes to get from A to B. NYC ↔ Frankfurt is 76ms great-circle. Light in fiber travels at about 2/3 of c, so the physical floor is 38ms one-way. You don't beat that without a lower-latitude path.

Jitter is the variance — how much that timing bounces around. P50 might be 80ms; P99 is 140ms; P99.9 might be 400ms because one packet hit a congested peering point and got buffered for 300ms. That tail is the part that kills you.

Solana propagates shreds — the little fragments a block gets chopped into for transmission — using Turbine, which is a unicast fan-out tree. The leader sends shreds to a small root set, they forward to their children, and the tree fans down from there. A single late shred at any layer delays everyone beneath it. The block can't be reconstructed until enough shreds arrive, and the next leader can't safely build on top until that happens. So P99 jitter at any intermediate hop compounds into P99+ jitter at the root of the next block.

The Malbec Labs team puts it bluntly in the Breakpoint 25 talk: "latency without control, you get a lot of jitter and it makes predictions that much harder." Their headline number is the worst-case time to land a transaction:

That's a 3× improvement at the tail. The median improvement is much smaller — DoubleZero can't move the speed of light. What it does is collapse the tail.

What DoubleZero actually is

Three layers, stacked. Real terminology this time, not marketing.

Physical layer. Network Bandwidth Contributors provide dedicated bandwidth with IPv4 connectivity and an MTU of 2048 bytes between two data centers — in practice, wavelength services running on existing long-haul fiber. Three flavors: L1 wavelength on DWDM/CWDM, L2 packet-switched VLAN extension, or L3 dedicated third-party bandwidth. Each contributed link terminates at a DZD (DoubleZero Device — a physical switch, in practice a pair of Arista 7280CR3As plus AMD V80 NICs, sitting in a 4U/4KW rack (a fridge-sized rack drawing 4 kilowatts) at each end).

Exchange layer. Contributor links bridge to each other at DZXs (DoubleZero Exchanges) — interconnect points in major metros. This is what turns N point-to-point contributions into an actual mesh.

Software layer.

• Controller — derives device configuration from on-chain state.
• Config Agent + Telemetry Agent — run on each DZD. Config Agent applies whatever the Controller says; Telemetry Agent measures latency, jitter, and packet loss via TWAMP — a network round-trip latency measurement protocol — and publishes results.
• doublezerod — the daemon that runs on the validator or RPC host. Manages a doublezero0 tunnel interface, the routing table, and the BGP (the internet's routing protocol) session into the mesh.
• On-chain ledger — serviceability state and telemetry get written to a Solana program. Network state is verifiable; routing isn't a black box.

What it isn't: it's not "BGP-free." DoubleZero actually uses BGP — the internet's routing protocol — but inside its own permissioned mesh, on 169.254.0.0/16 link-local addresses over GRE (Generic Routing Encapsulation, a packet-tunneling protocol; IP protocol 47), with every peer and policy known. The difference from the public internet isn't "no BGP," it's "BGP across a deterministic mesh of N participants" versus "BGP across tens of thousands of unknown ASes doing best-effort policy routing." Same protocol, completely different blast radius.

The live multicast group state, contributor list, and per-link telemetry are all published on a public dashboard — including a dedicated view just for multicast topology. The transparency is by design: contributors get paid based on measured service quality, so the measurements have to be public.

The multicast trick

Here is what makes DoubleZero structurally different from "we leased some fiber."

Turbine sends shreds as unicast — one packet to one recipient at a time. The leader picks a root validator and sends one copy. The root forwards a copy to each of its children. Each child forwards to its children. To deliver one block to 1500 validators, the network ends up carrying 1500 copies of every shred at the worst layer, and the depth of the tree piles on latency at every hop.

The public internet has no choice but unicast at that scale. IP multicast — one send that fans out to many receivers — exists in the spec, but it's almost universally turned off across the open internet, because no ISP wants to carry replicated traffic for a third party. So Solana built a unicast tree on top of best-effort unicast delivery.

DoubleZero adds native multicast as a connection mode. A publisher sends one copy of a packet to a multicast group. The mesh then replicates that packet right in the switch fabric — at each DZD, in hardware. The fan-out happens once and the copies go to every subscriber on the right egress links. From the publisher's side, one send. From each subscriber's side, one receive at near-line-rate.

For shred propagation specifically, the talk puts a number on it: Frankfurt subscribers saw gains exceeding 16ms on shred delivery via multicast vs unicast. That's per hop, per shred. For a hot validator on the next leader's slot, 16ms is 4% of the entire slot window.

  ┌─ PUBLIC INTERNET — UNICAST TURBINE ──────────────────────────────┐
  │                                                                   │
  │   leader ──► root ──► child ──► grandchild ──► …                  │
  │                                                                   │
  │   N copies of every shred · N hops of compound jitter             │
  │                                                                   │
  └───────────────────────────────────────────────────────────────────┘

  ┌─ DOUBLEZERO — NATIVE MULTICAST ──────────────────────────────────┐
  │                                                                   │
  │   publisher ─► group ──────╮                                      │
  │                            │   (1 copy in, hardware replicated    │
  │                       ┌────┴────┐    at each DZD switch fabric)   │
  │                       ▼    ▼    ▼                                 │
  │                     sub  sub  sub  …                              │
  │                                                                   │
  │   1 send · 1 receive per subscriber · sub-ms jitter               │
  │                                                                   │
  └───────────────────────────────────────────────────────────────────┘

You can run Turbine over DoubleZero unicast and still win on jitter. Or you can run shred distribution over DoubleZero multicast and win an extra 16+ms per hop on top. The multicast path is the harder integration — and the bigger payoff.

Connecting a validator

The part people get wrong: the original DoubleZero pitch sounded like "lease a wavelength and rebuild your validator." The actual story is much smaller than that.

The killer feature is IBRL — Increased Bandwidth, Reduced Latency, a connection mode that lets validators and RPC nodes connect to DoubleZero without restarting their blockchain clients. You don't fork Agave or Firedancer. You don't take downtime. doublezerod brings up a tunnel interface and a routing table, and the validator's existing sockets just start using the better path for any peer that's also on the mesh.

Prerequisites are unglamorous:

• x86_64 server, Ubuntu 22.04+ / Debian 11+ / Rocky / RHEL 8+, installed directly on the validator host (not in a container)
• Public IP, no NAT
• Firewall opens IP protocol 47 (GRE) and TCP/179 (BGP) on the link-local range
• A cross-connect from your colo cage to a participating DZX

# Ubuntu mainnet-beta install
curl -1sLf https://dl.cloudsmith.io/public/malbeclabs/doublezero/setup.deb.sh | sudo -E bash
sudo apt-get install doublezero

doublezero keygen        # generates identity at ~/.config/doublezero/id.json
doublezero address       # prints your DZ pubkey
sudo systemctl status doublezerod
doublezero latency       # discover DZDs, measure ping
doublezero status        # confirm connection

doublezero latency is the one to run before you sign anything. It lists the DZDs you can reach and shows their ping times — if your nearest POP is more than a few ms away, the cross-connect topology isn't right yet.

Testnet and mainnet-beta are physically distinct networks, not the same fiber with different routing. You pick one at install time, via the repo URL.

What does the validator binary see? Nothing changed — except that traffic to peers also on the mesh now exits doublezero0 instead of the default route, with much tighter jitter. Turbine and gossip just pick paths from the OS routing table; they don't know or care that the better path happens to run on dedicated fiber.

Connecting a searcher or extractor

The geometry is different for a searcher — the MEV bot operator trying to get a transaction in front of the right validator. You don't need to talk to every validator. You need to talk to the current leader, and the next few leaders, with the lowest possible tail latency. The leader schedule is public.

Today's serious-MEV stack looks like this: co-locate near a major validator concentration (FRA, NYC, TYO), stream tx submissions to multiple validator vote accounts, optionally route through Jito's relayer for bundle semantics, and pray the path to this slot's leader isn't going through Ashburn on a bad afternoon.

With a DoubleZero port, your tx leaves your host, takes one mesh hop to the leader's POP, and lands in their ingress queue in deterministic time. The variance just disappears.

Per-millisecond revenue for searchers is asymmetric — lose the race on a profitable opportunity and you lose the entire opportunity, not a fraction of it. So the math for paying for a DZ port works out at much smaller scales for searchers than it does for validators.

Compared to renting your own fiber

HFT firms have been solving this problem since 2010, and the comparison is worth spelling out, because it's exactly why DoubleZero is structured as a shared substrate.

The HFT model. A trading firm signs a dark-fiber lease (or a microwave path, McKay/Anova-style) on one specific route — say NYC → Aurora for CME, or NYC → London for LSE. They light their own wavelength, terminate at colocation cages on both ends, and own every meter exclusively.

You get the absolute best latency on earth for that route. You also pay for the entire pipe whether or not you ever fill it. For a firm doing $100M/year of MEV, that math works. For a 50-validator cluster trying to recover skipped-slot APY, it doesn't.

The DoubleZero model. A bandwidth contributor dedicates a wavelength to the shared substrate, terminates it on a DZD pair (Arista 7280CR3A + AMD V80, 4U/4KW per side), and bridges to the rest of the network at the nearest DZX. Lots of validators and searchers share ports into that substrate. Capacity is multiplexed, telemetry is published, and contributors get paid via the on-chain Shapley-value reward program — Shapley value being a fair-split formula from game theory — in proportion to their measured contribution to network quality.

The multicast row is the interesting one. Even if a validator could afford an exclusive dark-fiber lease, they'd still only get unicast out of it — multicast at scale needs a coordinated mesh where every switch knows the group membership. That's something a shared network can do and a per-firm lease structurally can't.

Frankendancer and the integration surface

This is the part that gets undersold in the Malbec docs, and it's the most interesting bit for anyone who actually writes validator code.

Frankendancer is on mainnet, and the wins compound with DoubleZero. Frankendancer — the production hybrid where Firedancer's networking and block-packing replace Agave's, then Agave's runtime does the actual execution — has been on mainnet since September 2024. It posts roughly 22% faster shredding without Merkle trees and almost 2× faster with Merkle trees versus Agave's path on the same hardware. That's a pure CPU-side win; the wire is still the same wire.

Drop Frankendancer on a DoubleZero port and the two improvements stack. Frankendancer takes less time to produce the shreds; DoubleZero takes less wall-clock and far less jitter to deliver them. Inside a 400ms slot, the leader claws back time on both ends — more time to keep the tx ingress open, less time spent waiting for the previous block's shreds to fully arrive before it can start building.

The integration shape is the net_tile. Firedancer's net_tile is the per-CPU thread that owns one NIC — a tile being a pipeline stage, one unit of the validator's work. The natural place to wire in DoubleZero is to bind a specific net_tile to the doublezero0 interface — consensus traffic to peers also on the mesh goes through that tile, and everything else goes through a separate net_tile bound to the regular uplink. Because affinity is at the tile level, you can pin DoubleZero traffic to the cores closest to its NIC's PCIe slot, with no contention from public-internet traffic on the same thread.

The upstream net_tile patch that ties Firedancer's userspace directly to doublezero0 isn't public yet — but pieces of the integration are already shipping. Three layers of it to watch:

The kernel-side plumbing is already shipping. The tile-side glue and a packaged MEV-fork variant are the two missing pieces — both mechanically obvious, neither published yet.

What developers can build on top

The public surface as of mainnet-beta includes a few pieces worth knowing about:

• doublezero-solana Rust crate (v0.3.0 as of late 2025) — the primary client integration. If you're writing a Solana service that wants to be DZ-aware, this is what you depend on.
• doublezero-edge-solana shred receiver example — a minimal program that subscribes to shreds over DoubleZero and prints throughput. About as close to a "hello world" for the multicast subscription API as you'll get.
• Package sidecar — polls a release feed for new doublezero versions and runs apt-update. Boring infra, but the kind of thing you want pinned and audited if you're running validators at scale.
• On-chain DZ ledger program — telemetry and serviceability state get written to a Solana program. Anyone can build a third-party telemetry consumer without asking permission; you just need RPC access.
• TWAMP telemetry feed — the same data that drives contributor payouts. Public, structured, and queryable from the public telemetry dashboard.

So, concretely, here's what a competent dev could build this quarter without asking anyone's permission:

Who's this for? Mostly Solana validator operators, MEV searchers, oracle providers, and RPC providers — anyone whose business is denominated in tail-latency reduction. Secondary audience: anyone running a private financial protocol with latency-sensitive participants (perp DEXes with their own market-maker programs, orderbook venues with non-validator participants who still care about tx submission speed). Tertiary: blockchain projects beyond Solana — DZ's transport layer is chain-agnostic, and Malbec has been clear that the network isn't Solana-exclusive in principle.

Performance, with real numbers where they exist

Three things to measure, in order of how much they matter to participants:

Worst-case tx land time. Malbec's published figure: 500ms on DoubleZero vs >1.5s on public internet at the tail. This is the headline number from the BP25 talk, and the one most likely to move both validator skip rate and searcher inclusion rate.

Multicast vs unicast shred delivery. Frankfurt subscribers saw a >16ms improvement per delivery in published testing. For a leader sitting on a 400ms slot, that's 4% of the entire window recovered — per shred, per hop. Compound that across Turbine depth and the slot window loosens meaningfully, which lets the leader keep tx ingress open longer without risking a skip.

Real-time link-level telemetry. TWAMP measurements between every DZD pair are published to the on-chain ledger and visible on the public telemetry dashboard. That's unusual — most private networks publish nothing — and it's load-bearing for the whole economic model. Contributors get paid based on what TWAMP says, so the measurements are the asset.

For the median validator, the win is less dramatic than the tail number suggests. Their path on public internet is fine most of the time. The pitch for joining is the tail: P99 jitter and skip rate during congested hours, especially when the leader schedule puts a Frankfurt validator right after a Tokyo validator and the trans-Pacific shred propagation has to land in 200ms.

The outer ring — where DoubleZero quietly uses FPGA

Most of DoubleZero is fiber, switching, and userspace daemons. There's one place where it does reach for silicon: the outer ring. Before a packet hits a subscribing validator's NIC, it crosses a DZ edge node running an FPGA pre-gossip filter — line-rate inspection that drops duplicate, malformed, and obviously-low-quality transactions before they ever touch the inner ring. The validator never sees the noise; its verify and dedup tiles get a cleaner stream to chew on.

This is the FPGA use-case the full-FPGA-validator math couldn't justify, finally landing where it actually makes sense: a single stream, one job, one card, sitting on the network side of PCIe. The filter isn't doing validation, it isn't executing transactions, and it isn't holding any of the validator's state — all the un-FPGA-friendly work stays on the CPU where it belongs. The hardware shows up exactly at the seam where stream-friendly work meets the rest of the pipeline, which is precisely what tile-style architectures were designed to allow.

Phase II — paying validators to leave Frankfurt

The geographic concentration problem is older than DoubleZero. Cheap European bare-metal pulled the early Solana validator set into Frankfurt and a few neighboring cities. A 2026 stake snapshot still showed central Europe holding the plurality of stake — which is a structural disadvantage for users on every other continent.

On March 9, 2026, DoubleZero launched its Phase II delegation program: 2.4 million SOL redirected out of a 13M-SOL pool toward validators operating in São Paulo, Singapore, Hong Kong, and Tokyo — explicitly underrepresented regions. The idea is to hand a stake premium to operators who relocate or open new sites, while the fiber side simultaneously strips away the latency tax that historically made those locations uneconomic. The two mechanisms come at the same problem from opposite directions.

The funding model is simple: validator operators pay a 5% "seat fee" on their validator-side income (inflation and block rewards, MEV excluded), which is then either paid out to DoubleZero contributors pro-rata to measured network quality, or burned. It's the same shape as the Solana validator commission curve — a small slice of a recurring revenue stream — applied to a piece of infrastructure that didn't previously have one.

What DoubleZero does not fix

The honest flip side of the "what it unlocks" list is what stays unsolved. DoubleZero is a network-layer product, and here's the map of what that does and doesn't cover:

The cleanest way to think about it: DoubleZero and Firedancer are complementary, not competitive. One optimizes the network path between validators, the other optimizes the work each validator does. Alpenglow sits on top of both and asks them to cooperate.

What this unlocks, and the counter-thesis

If this works at scale, three things happen:

1. Skip rate compresses geographically. A Tokyo validator and a Frankfurt validator stop paying a permanent tax just for being far from the consensus center of mass. The leader schedule becomes location-agnostic in practice.
2. The Jito relayer becomes one option among several. It still owns bundle semantics and tip routing. What it stops owning is "lowest jitter to the leader," because the underlying network already gives you that.
3. Programmable network primitives become possible. Native multicast is just the first one to ship. In-network ordering, hardware-timestamped attestations, and deterministic shred reconstruction are all things a private substrate can do and the public internet can't.

The counter-thesis is worth saying out loud. Maybe Solana ships protocol-level improvements that close the jitter gap without anyone needing private fiber. Async block production, better Turbine fanout, QUIC tuning, smarter repair — each one chips at the same problem from inside the protocol. If the median validator's skip rate gets down to 0.3% on plain internet, the marginal case for DoubleZero on the validator side gets weaker.

The MEV/searcher side is more durable. Tail-latency for tx submission is going to matter no matter what happens inside consensus, and the multicast primitive can't be reproduced at the protocol layer — it needs the network layer to cooperate.

The bet Malbec is making is that both fronts move — protocol improvements happen, AND a dedicated network is still worth it for the participants who care most. That's a reasonable bet, because the participants who care most are exactly the ones with the budget to pay for it. The network has a real two-sided business model: validators pay to capture skipped-slot APY, searchers pay to capture inclusion-rate edge, and contributors get paid in proportion to measured quality.

40% of Solana validators have already joined a six-month-old network. That is the strongest evidence the bet is paying.

What to watch over the next year

• Skip rate distribution by region. Tokyo/Singapore validators on DoubleZero converging with Frankfurt validators means the thesis is working. Track it via standard validator leaderboards plus the public DZ telemetry dashboard.
• Multicast group adoption. Visible on the public multicast-groups view of the same dashboard. The curve to watch is how many validators move from unicast-over-DZ to multicast-over-DZ.
• Jito relayer's share of bundle flow. If it drops as searcher DZ ports proliferate, the unbundling is happening.
• Contributor growth and POP coverage. Each new DZX in a new metro pulls in the validators racked nearby. Network effects compound.
• Firedancer's net_tile integration. Clean tile-level integration with doublezero0 is the obvious next milestone. The first public patch is the signal to watch.

The simplest test: in twelve months, look at the top 50 validators by stake and count how many have a DoubleZero port. If it's most of them, the bet paid off. If it's a handful, public internet was good enough after all. Right now the answer is somewhere in the middle, and rapidly climbing.

References