Fire-Flyer AI-HPC

TL;DR#

• What was built: Fire-Flyer 2, a cluster of 10,000 PCIe A100 GPUs (not the NVLink-switched SXM parts in NVIDIA's DGX boxes). It has about 1,250 compute nodes. Each node has 8 GPUs and a single 200 Gbps InfiniBand NIC (§III-A). It was designed and built from 2021, and the paper reports four years of design, construction and operation.
• Cheap network: two separate two-layer fat-tree "zones" joined by a few links, carrying both compute and storage traffic. It needs 122 switches, versus 1,320 for an equivalent DGX-style three-layer fat-tree (Table III).
• Software makes up the gap:
• HFReduce does gradient allreduce on the CPU, beating NCCL on this hardware: 6.3–8.1 GB/s vs 1.6–4.8 GB/s (§IV).
• HaiScale is the parallel-training library.
• 3FS is a distributed file system built for SSDs and RDMA (8 TB/s read).
• HAI Platform handles scheduling and fault recovery.
• Headline claim: about 80% of DGX-A100 performance at about 50–60% of the cost, with about 40% less energy (§III-C, Table II, §X).
• Operations data: a year of GPU fault statistics. NVLink-bridge errors (Xid 74) are 42.57% of all logged Xid errors (Table VI).

Why it matters#

Most LLM papers treat the cluster as a given. This one explains the cluster itself and why it looks the way it does. High-Flyer, the quantitative fund that DeepSeek grew out of, built Fire-Flyer 2 in 2021 for general deep learning. The key early bet was that most workloads (ResNet, BERT, Mask-RCNN) have under 1B parameters and don't need the expensive all-to-all NVLink fabric of a DGX server. PCIe A100s were cheaper and used less power. They were weaker at communication, so the team wrote their own communication, storage and scheduling software to make up for it.

When LLMs arrived, they added NVLink bridges between pairs of GPUs. The cluster then trained DeepSeek's models; the acknowledgments say it supports the training of the DeepSeek-AI LLM series. That puts it underneath DeepSeek LLM, DeepSeekMoE, DeepSeek-V2 and the rest of the arc, although DeepSeek-V3 reports training on a separate 2,048-GPU H800 cluster.

Key idea Buy cheaper, weaker hardware and spend engineering effort on software that routes around its specific bottlenecks. HFReduce moves allreduce to the CPU because PCIe peer-to-peer is slow on these servers. Traffic classes are isolated by hand because adaptive routing spread congestion and DCQCN could not be tuned for mixed traffic.

The same habit appears throughout the series. Find the binding constraint of the hardware, then design around it. DeepSeek-V3's DualPipe, node-limited routing and FP8 tricks apply this to H800s. The "next-generation" multi-plane network sketched here in §IX shows up again in DeepSeek's 2025 hardware retrospective (arXiv 2505.09343).

Background you need#

• Allreduce: in data-parallel training, every GPU computes gradients on its own data, and the gradients must be summed across all GPUs every step. Allreduce does this. NCCL, NVIDIA's collective library, implements it with ring or tree algorithms that run as GPU kernels.
• PCIe vs SXM/NVLink: SXM A100s in a DGX connect all 8 GPUs through NVSwitch at 600 GB/s. PCIe A100s talk to each other and to the network card through the CPU's PCIe lanes, which are much slower. An optional NVLink bridge links only a pair of cards.
• Fat-tree topology: a switch network in which leaf switches connect to servers and spine switches connect leaves. It provides full bisection bandwidth: any half of the machines can talk to the other half at full speed. Each extra layer multiplies the number of ports supported but adds many switches.
• RDMA and InfiniBand: Remote Direct Memory Access lets one machine write into another's memory without involving its CPU. InfiniBand (IB) is the RDMA network used here. GDRCopy is an NVIDIA library for fast, low-latency CPU access to GPU memory.
• Parallelism strategies: data parallel (DP), pipeline parallel (PP), tensor parallel (TP), expert parallel (EP), and ZeRO/FSDP sharding of optimizer state and parameters (ZeRO, arXiv 1910.02054). §II-B of the paper defines all of these briefly.
• Xid errors: numbered fault codes the NVIDIA driver writes to the system log, e.g. Xid 74 for NVLink errors and Xid 63/64 for memory ECC events.

The core ideas#

1. The node and the network (§III)#

The node (Table I, Figure 4).

Each GPU and the NIC attach directly to the CPU with no PCIe switch, and the NIC sits on its own PCIe root complex so it doesn't contend with GPUs. The authors argue that one NIC per 8 GPUs was enough for their storage and communication needs at the time. This choice later constrains the design (see §IX below).

Cost-performance (Table II).

The network (§III-B, Figure 5). A two-layer fat-tree built from 40-port, 200 Gbps Mellanox QM8700 switches tops out at 800 ports (20 spine + 40 leaf switches). Ten thousand GPUs need about 1,250 compute nodes plus storage servers, which is too many for one tree. The obvious fix is a third switch layer. Instead, they build two zones, each an 800-port two-layer fat-tree serving about 600 compute nodes.

• Storage is shared: each storage server has two NICs, one into each zone, so every compute node can reach all storage.
• Zones are lightly linked: the two zones are joined by only a few links. The scheduler allows at most one job to span both zones, and in that topology only one pair of nodes communicates across the boundary.

Switch counts (Table III, §III-C).

The prices are relative units. The two-zone design saves 40% on networking versus a comparable three-layer tree. Overall, Fire-Flyer 2 costs about half (11,600 / 23,000) of a DGX-based build. The authors also chose fat-tree over Dragonfly, which is comparably cheap but lacks the bisection bandwidth that mixed storage and compute traffic needs. They chose IB over RoCE (RDMA over Ethernet) because RoCE was less mature in 2021.

2. HFReduce: allreduce through the CPU (§IV)#

The problem. On these servers NCCL allreduce was slow, about 4 GB/s. The authors traced the root cause with AMD and NVIDIA engineers. EPYC Rome CPUs lack a PCIe "chained write" feature that speeds up direct GPU↔NIC transfers, capping GPU-to-NIC bandwidth at about 9 GiB/s (§IV-D2).

The fix. Avoid GPU↔NIC peer-to-peer altogether:

1. Device-to-host: each GPU asynchronously copies its gradient chunks to CPU memory, using GDRCopy for small transfers and cudaMemcpyAsync for large ones.
2. Intra-node reduce: the CPU sums the 8 GPUs' chunks with SIMD vector instructions. FP32, FP16, BF16 and FP8 are supported.
3. Inter-node allreduce: CPUs across nodes run a double binary tree allreduce (the algorithm NCCL also uses) over RDMA writes, pipelined in chunks.
4. Host-to-device: the result is copied back. With GDRCopy, data read once from host memory sits in CPU cache and is written to the 4 GPUs on the same NUMA node, cutting host-memory reads 3× versus cudaMemcpyAsync.

Why this beats NCCL here (§IV-B).

• Less PCIe traffic. In a ring over \(n\) GPUs, each unit of data makes \(2n-1\) hops, each using one GPU's inbound and another's outbound PCIe bandwidth. That is about \((2n-1)/n \approx 2\) units of PCIe bandwidth per unit of data. HFReduce does one device-to-host and one host-to-device copy, using one unit.
• No GPU kernels. Copies run on the GPU's copy engines, so allreduce does not occupy SMs or slow down the backward pass running alongside it.

Result (Figure 7a). For a 186 MiB allreduce, HFReduce reaches 6.3–8.1 GB/s of inter-node bandwidth versus 1.6–4.8 GB/s for NCCL, scaling from 16 to 1,440 GPUs.

Its own ceiling (§IV-D3). Count host-memory traffic per byte of gradient:

That is 24× the data size. At a practical 320 GB/s of memory bandwidth, the ceiling is about 13.3 GB/s, or about 12 GB/s after algorithm and network overheads. The measured figure is about 8 GB/s. The gap comes from another EPYC quirk: GPU5 and GPU6 share one PCIe root port, whose link to the CPU's internal bus caps at about 37.5 GB/s, and it gets worse under bidirectional traffic.

HFReduce with NVLink (§IV-C). Once NVLink bridges were added, each GPU pair first reduces over NVLink, so less data crosses PCIe to the CPU. After the CPU stage, each GPU of the pair receives half the result and they all-gather it over NVLink. This exceeds 10 GB/s of inter-node bandwidth (Figure 7b).

3. HaiScale: the training library (§V)#

HaiScale provides DDP, FSDP, PP, TP and EP on top of HFReduce. The PCIe-specific tricks:

• DDP with asynchronous allreduce. Because HFReduce uses no GPU compute, gradient allreduce overlaps fully with backprop. VGG16 trains in half the time of PyTorch DDP with NCCL, with about 88% scaling efficiency from 32 to 512 GPUs (Figure 8a).
• Pipeline parallelism with one NIC. All 8 GPUs in a node share one NIC, so simultaneous PP sends collide. HaiScale assigns the 8 GPUs of a node to different DP ranks, which staggers their PP communication in time. LLaMA-13B step time falls from 64.118 s on 64 GPUs to 9.717 s on 512, a 91% parallel efficiency (Figure 9a).
• MoE. DeepSeekMoE-16B falls from 79.615 s per step on 40 GPUs to 10.71 s on 320 (92.92% efficiency) and 6.535 s on 640 (76.14%) (Figure 9b, §V-B2).
• FSDP. Reduced memory fragmentation, and allgather/reduce-scatter overlapped with compute. The optimizer step is split into the backward pass. This gives 95% scaling from 16 to 128 GPUs on GPT2-Medium and nearly half the training time of PyTorch FSDP (Figure 8b).
• Tensor parallelism is only practical within an NVLink-bridged GPU pair (§V-B1).

4. Keeping a shared network uncongested (§VI-A, §VIII-A)#

Compute and storage share one fabric, so heavy storage reads could stall gradient sync. Four measures address this:

1. Traffic classes. HFReduce, NCCL, 3FS storage and other traffic each get an IB Service Level mapped to separate Virtual Lanes, with configured bandwidth shares. One class cannot block another; this avoids head-of-line blocking.
2. Static routing. Adaptive routing made congestion spread worse under storage incast (many senders to one receiver). They use static routes instead, with storage, compute and management nodes placed so storage traffic spreads evenly over leaf→spine links.
3. NCCL tuning. NCCL is routed through the NIC and GPUs on the same NUMA node, and PCIe Relaxed Ordering is enabled.
4. Application-level flow control in 3FS (below).

They also disabled DCQCN, IB's standard congestion-control algorithm. They could not find parameters that served both HFReduce and 3FS traffic, and relied on the measures above instead (§VIII-A).

5. 3FS, checkpointing and scheduling (§VI-B, §VI-C, §VII-A)#

3FS hardware (Table IV). 180 storage nodes, each with 16 PCIe 4.0 NVMe SSDs (15.36 TB each) and 2 × 200 Gbps IB NICs. In total that is 2,880 SSDs and over 20 PiB of mirrored storage. Theoretical read bandwidth is 9 TB/s, and 8 TB/s was achieved.

Design.

• Metadata: inodes and directory entries are stored as key-value pairs in a distributed KV store. Several meta services run at once, and all their state is persisted in that store.
• Data: files are split into chunks, each replicated along a chain of storage targets using CRAQ (Chain Replication with Apportioned Queries). Writes go down the whole chain; reads can come from any replica, so all SSDs serve reads.
• Incast control: a storage server reads data from SSD, then asks the client for permission before sending. The client limits how many servers send at once. This adds latency but keeps throughput sustainable.
• 3FS-KV: a key-value, message-queue and object store built on 3FS. It backs DeepSeek's KV Context Caching on Disk for LLM serving.

Checkpoint manager (§VII-A). Parameters and optimizer state are copied asynchronously to host memory, then written to 3FS in chunks at over 10 GiB/s per node. Saves take seconds and happen about every 5 minutes, so a failure loses at most about 5 minutes of work.

HAI Platform (§VI-C). A time-sharing scheduler. Jobs must handle an interrupt signal: checkpoint, report, then resume later from the checkpoint. The scheduler can therefore pre-empt and repack jobs at node granularity. The authors report about 99% utilization. It is open-sourced at github.com/HFAiLab/hai-platform.

6. Failures in practice (§VII-B, §VII-C, Appendix)#

Validator. A weekly suite checks:

• link speeds and status;
• CPU stress and memory bandwidth;
• every byte of GPU memory;
• full-memory GEMMs that catch logic faults;
• intra-node allreduce, which exercises NVLink;
• storage bandwidth.

Failing nodes are removed from scheduling. The authors note they have hit GPU compute and memory errors not caught by ECC that showed up as gradient-norm spikes, loss explosions or non-convergence. Catching bad nodes before training reaches them matters.

Xid statistics over one year (Table VI), 12,970 errors total.

Two more observations. GPU ECC faults far outnumber CPU memory ECC faults (Figure 10). IB link "flash cuts" (brief link drops) are 30% of hardware faults once Xid 74 is excluded, and occur randomly over time (Figure 11). For comparison, the authors cite another study where NVLink failures were 52.42% of failures (§VIII-D).

7. What comes next: designing for MoE (§IX)#

MoE training is dominated by all-to-all traffic, and one NIC per 8 GPUs does not support it well. The proposed next-generation node keeps PCIe GPUs but goes to a 1:1 GPU-to-NIC ratio, like DGX-H100/B100. The NICs connect to separate fat-tree "planes", forming a multi-plane network (Figure 12). With 128-port 400 Gbps RoCE switches, a 4-plane two-layer fat-tree could reach 32,768 GPUs. Moving from IB to RoCE switches would cut network cost further.

Key numbers#

How to read the paper#

The paper is in IEEE two-column format (SC24). Several tables are placed far from where they are cited, so expect to flip back and forth.

• I. Introduction — Skim. The bullet list of topics is the paper's map.
• II. Background (A–D) — Skip on first pass if you know DP/PP/TP/EP/ZeRO. Figure 2 (compute vs memory/interconnect growth) is the one useful chart.
• III. Fire-Flyer 2 (A–C) — Read closely. Tables I–III and Figures 4–5 carry the whole cost argument.
• IV. HFReduce (A–D) — Read closely. Algorithms 1–2, Figure 7, and especially IV-D. The bottleneck accounting (24× memory traffic, root-port limits) is the best systems reasoning in the paper.
• V. HaiScale — Read V-B (PP with a single NIC, Figure 9). Skim V-A and FSDP.
• VI. Advanced co-design — Read VI-A (congestion measures). Read VI-B on 3FS if you care about storage; the CRAQ and request-to-send paragraphs are the substance. Skim VI-C.
• VII. Stability and Robustness — Read VII-C and Table V (Xid taxonomy). Skim the checkpoint manager and validator.
• VIII. Discussion — Read VIII-A (why DCQCN was disabled) and VIII-B (why NVLink was added late). Skim C–D.
• IX. Future Work — Read (short). It links directly to how DeepSeek later built for MoE.
• X. Conclusions — Skim.
• Appendix tables VI–VIII — Skip unless you want the raw failure counts.

Suggested order: III → IV → V-B → VI-A → VII-C → VIII-A/B → IX. About 1–1.5 hours.

Check your understanding#

Q1Why does moving allreduce onto the CPU help on these servers, when CPUs are much slower than GPUs at arithmetic?

The bottleneck is data movement, not arithmetic. EPYC Rome lacks PCIe chained writes, so direct GPU↔NIC transfers cap at about 9 GiB/s, and NCCL's ring uses about twice the PCIe bandwidth per byte. HFReduce copies each byte to the host once and back once. It sums with SIMD and talks to the NIC from host memory, which avoids both problems. It also uses no SMs, so it overlaps with backprop.

Q2HFReduce's theoretical ceiling is about 13 GB/s. Where does that number come from, and why is the measured figure lower?

Per byte of gradient, the node performs about 24 host-memory reads and writes across the device-to-host, reduce, network and host-to-device phases. Dividing a practical 320 GB/s of DDR4 bandwidth by 24 gives about 13.3 GB/s. Measured throughput is about 8 GB/s because two GPUs share a PCIe root port capped near 37.5 GB/s to the CPU, and bidirectional transfers reduce that further.

Q3Why split the cluster into two fat-tree zones instead of building a three-layer fat-tree?

A two-layer tree of 40-port switches maxes out at 800 ports, too few for about 1,250 compute nodes plus storage. A third layer would need many more switches and optics. Two zones of about 600 nodes each, with shared dual-homed storage and only a few inter-zone links, cost 122 switches. The price is that at most one job may span zones, a restriction the scheduler enforces.

Q4Why did the team disable DCQCN congestion control and avoid adaptive routing?

Compute (HFReduce) and storage (3FS) share one fabric and behave very differently. DCQCN could not be tuned to suit both. Adaptive routing spread congestion further under storage incast. The team isolated traffic classes on separate IB virtual lanes, used static routes with careful node placement, and added request-to-send flow control in 3FS.

Q5How does HaiScale make pipeline parallelism work with only one NIC per 8 GPUs?

If the 8 GPUs of a node all send pipeline activations at the same moment, they fight over the single NIC. HaiScale assigns them to different data-parallel ranks, which staggers the timing of their pipeline communication. LLaMA-13B reaches 91% parallel efficiency from 64 to 512 GPUs this way.

Limitations and open questions#

• The performance comparison is narrow. The "80% of performance" figure is per-GPU GEMM throughput (Table II), not end-to-end training throughput measured on both systems. Cost figures are relative units without absolute prices.
• Weak for communication-heavy workloads. One NIC per 8 GPUs and pair-only NVLink limit tensor and expert parallelism. The authors acknowledge this in §IX by moving to 1:1 GPU:NIC for MoE. DeepSeek-V3 was trained on an H800 cluster, not on Fire-Flyer 2.
• Highly tuned to specific hardware. HFReduce's advantage comes from specific EPYC PCIe limits. On CPUs with chained writes, or on NVSwitch systems, NCCL's GPU-direct path may win. The paper notes that EPYC Genoa still has the host-bridge limit.
• NVLink bridges are fragile. They cause the largest share of Xid errors (42.57%), so the cheap way to add NVLink has a reliability cost.
• Some numbers are approximate or inconsistent. Storage is "nearly 200 servers" in §III-B but 180 nodes in §VI-B. Cost savings are stated as "half" in one place and "60% of the cost" in another. Several scaling results use small models (VGG16, GPT2-Medium).
• Limited reproducibility. 3FS, HFReduce and HaiScale were not public when the paper appeared; only HAI Platform was linked. 3FS was later open-sourced by DeepSeek.

Further reading#

• Later in the series: DeepSeek-V3 applies the same co-design approach to H800s: custom all-to-all kernels, DualPipe and FP8, plus its own hardware wishlist. DeepSeekMoE is the 16B MoE model benchmarked in §V-B. DeepSeek LLM and DeepSeek-V2 were trained in this period.
• Background papers: ZeRO (arXiv 1910.02054); PyTorch FSDP (arXiv 2304.11277); Zero Bubble pipeline parallelism (arXiv 2401.10241); CRAQ, "Object Storage on CRAQ" (Terrace & Freedman, USENIX ATC 2009).
• Follow-up: Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, ISCA 2025 (arXiv 2505.09343), which discusses a multi-plane network topology.
• Code: HAI Platform at github.com/HFAiLab/hai-platform, as given in the paper. 3FS was later released at github.com/deepseek-ai/3FS.