主流序列并行(SP)方案对比 + 大模型通信原语

这篇笔记用图文 + 动图把 主流序列并行(SP)方案 串起来:Megatron-SP、DeepSpeed-Ulysses、Ring-Attention / Context Parallel、以及混合的 USP,逐个讲机制、配动图、对比优缺点;最后讲清 主流大模型(含 DeepSeek MoE)用的通信原语。This note walks through the mainstream sequence-parallel (SP) schemes with figures and animations: Megatron-SP, DeepSpeed-Ulysses, Ring-Attention / Context Parallel and the hybrid USP — mechanism by mechanism, each with a figure, then a side-by-side comparison. It closes with the communication primitives the major LLMs actually use, DeepSeek MoE included.

一句话总览: 先分清"SP"的两层意思 —— ① Megatron-SP 是 TP 的补充,省非 TP 区的激活显存;② 长上下文 SP(Ulysses / Ring / USP)才是把序列维切到多卡、训超长序列。底层都靠 集合通信原语(NCCL/HCCL):AllReduce、AllGather、ReduceScatter、All-to-All、P2P。In one line: first separate the two meanings of "SP". ① Megatron-SP supplements TP and saves activation memory outside the TP region; ② long-context SP (Ulysses / Ring / USP) is what actually shards the sequence dimension across GPUs to train very long sequences. Both rest on the same collective primitives (NCCL/HCCL): AllReduce, AllGather, ReduceScatter, All-to-All and P2P.

0 · 背景:为什么需要 SP0 · Background: why SP exists

自回归 Transformer 的激活显存随 batch × seq_len × hidden 线性增长,注意力更是接近平方级。序列推到 32K、128K、1M 时,单卡既放不下激活、也算不动注意力。SP 的核心思路:把序列这一维切到多张卡上,每卡只持有一段,从而摊薄显存与算力 —— 代价是引入跨卡通信。Activation memory in an autoregressive Transformer grows linearly with batch × seq_len × hidden, and attention closer to quadratically. Push the sequence to 32K, 128K or 1M and a single GPU can neither hold the activations nor compute the attention. SP's core idea: shard the sequence dimension across GPUs so each holds one slice, spreading both memory and compute — at the cost of cross-GPU communication.

1 · 先备齐"通信原语"1 · First, the communication primitives

所有并行策略最终都落到几种集合通信原语上(由 NCCL / 昇腾 HCCL 提供)。先看懂它们,后面的 SP 方案就是"用哪个原语、用几次"的区别。Every parallelism strategy ultimately reduces to a handful of collective primitives (supplied by NCCL or Ascend HCCL). Understand these and the SP schemes become a question of which primitive, and how many times.

原语Primitive	干什么What it does	谁在用Who uses it
AllReduce	各卡数据求和,结果广播回所有卡Sums data across GPUs and broadcasts the result back to all of them	DP 同步梯度、TPDP gradient sync, TP
AllGather	各卡的分片拼成全量,人手一份Concatenates each GPU's shard into the full tensor, one copy each	ZeRO/FSDP 取参数、TP+SPZeRO/FSDP parameter fetch, TP+SP
ReduceScatter	求和后再切片,每卡只拿一片Sums, then scatters so each GPU keeps only one slice	ZeRO/FSDP 归约梯度、TP+SPZeRO/FSDP gradient reduction, TP+SP
All-to-All	每卡把数据按目标重排发给所有卡(分布式转置)Each GPU redistributes its data to every other by destination (a distributed transpose)	Ulysses SP、MoE 专家并行Ulysses SP, MoE expert parallelism
P2P Send/Recv	点对点收发(All-to-All / Ring 的底层积木)Point-to-point send/receive (the building block under All-to-All and Ring)	流水线 PP、Ring SPPipeline PP, Ring SP

动图 · AllReduce:汇总求和 → 广播回所有卡Figure · AllReduce: sum at the centre, broadcast back to every GPU

▶ 四张卡的数据流向中心求和(Σ),再以扩散的脉冲广播回所有卡 —— 这就是 AllReduce▶ Four GPUs' data flows to the centre to be summed (Σ), then radiates back out to all of them — that is AllReduce

动图 · All-to-All:每卡把数据按目标重排发给所有卡(分布式转置)Figure · All-to-All: each GPU redistributes data by destination (a distributed transpose)

发送方Sender 接收方Receiver G0 G1 G2 G3 G0 G1 G2 G3 ▶ 每张卡都把自己的数据切成 4 份、分别发给 4 张卡(颜色=来源)。通信量低、与序列长度增长解耦,但并行度被 head 数限制 —— Ulysses 和 MoE 都靠它▶ Every GPU splits its data into four pieces and sends one to each of the four GPUs (colour = origin). Communication volume is low and decoupled from sequence growth, but the degree of parallelism is capped by the head count — both Ulysses and MoE rely on this primitive.

动图 · AllGather:每卡的分片,拼成人手一份的全量Figure · AllGather: each GPU's shard, assembled into a full copy for everyone

▶ 每卡原本只有自己那一片(实色),其余三片陆续"收齐"(渐入)→ 最终每卡都有全 4 片。ReduceScatter 是它的逆操作(先求和再各拿一片)▶ Each GPU starts with only its own shard (solid), then the other three arrive one by one (fading in) until every GPU holds all four. ReduceScatter is the inverse — sum first, then each takes one slice.

2 · Megatron-style SP(TP 耦合)2 · Megatron-style SP (coupled to TP)

这是最容易被混淆的"SP"。它不单独处理注意力的跨卡,而是作为 张量并行(TP)的补充:TP 已经把注意力/MLP 的权重按 head/维度切了,但 LayerNorm、Dropout、残差 这些区域 TP 没覆盖,激活仍是全量、冗余存在每张卡上。Megatron-SP 把这些区域的激活也沿序列维切开,从而省下这部分激活显存。This is the "SP" people most often confuse. It does not handle attention across GPUs at all; it acts as a supplement to tensor parallelism. TP already shards attention and MLP weights by head or by dimension, but LayerNorm, Dropout and the residual are not covered, so their activations stay full-size and redundantly replicated on every GPU. Megatron-SP shards those activations along the sequence dimension too, reclaiming that memory.

Megatron-SP:在 TP 区与 SP 区之间换原语Megatron-SP: swapping primitives at the TP/SP boundary

SP 区SP regionLayerNorm / Dropout激活按序列切activations sharded by sequence

⇄

边界BoundaryAllGather / ReduceScatter替代 TP 原来的 AllReducereplaces TP's original AllReduce

⇄

TP 区TP regionAttention / MLP权重按 head/维度切weights sharded by head / dimension

图 1 · Megatron-SP 把 TP 的一次 AllReduce 拆成 AllGather + ReduceScatter,顺手把非 TP 区激活也切了 —— 目标是省激活显存,不是训超长序列Figure 1 · Megatron-SP splits TP's single AllReduce into AllGather + ReduceScatter and shards the non-TP activations along the way. The goal is saving activation memory, not training longer sequences.

3 · DeepSpeed-Ulysses(all-to-all)

真正面向长序列的方案之一。平时每卡只持有一段序列(按 seq 切);一到注意力计算,用一次 All-to-All 把布局转置成"按 head 切"——于是每卡拿到完整序列、但只算自己负责的几个 attention head;算完再 All-to-All 转回去。One of the genuinely long-sequence schemes. Normally each GPU holds one slice of the sequence (sharded by seq); when attention runs, a single All-to-All transposes the layout to "sharded by head" — so each GPU now has the full sequence but only the attention heads it owns. After the computation another All-to-All transposes it back.

动图 · Ulysses:All-to-All 在「按序列切 ⇄ 按 head 切」之间转置Figure · Ulysses: All-to-All transposes between "sharded by sequence" and "sharded by head"

注意力之外:按序列切Outside attention: sharded by sequence

每卡 = 一段序列
(全部 head)each GPU = one sequence slice
(all heads)

⇄

All-to-All
转置

⇄

All-to-All
transpose

注意力之内:按 head 切Inside attention: sharded by head

每卡 = 完整序列
(部分 head)each GPU = the full sequence
(some heads)

图 2 · 每层注意力前后各做 All-to-All(共约 4 次)。通信省、实现简单,但并行度 ≤ head 数(GQA 下受 KV head 限制更紧)Figure 2 · One All-to-All before and after attention in every layer, roughly four in total. Communication is cheap and the implementation is simple, but the degree of parallelism is ≤ the head count (tighter still under GQA, where KV heads bind).

4 · Ring-Attention / Context Parallel(P2P 环)4 · Ring-Attention / Context Parallel (a P2P ring)

另一条长序列路线,也是超长上下文的主力。每卡持有序列的一个块的 Q/K/V;计算注意力时,把 K/V 块沿着卡组成的环依次传递,每收到一块就和本地 Q 算一次局部注意力,用 online-softmax(FlashAttention 思路)把分块结果累加起来 —— 通信可以和计算 overlap。The other long-sequence route, and the workhorse for very long context. Each GPU holds the Q/K/V of one block of the sequence; to compute attention, the K/V blocks are passed around a ring of GPUs. Each time a block arrives it is combined with the local Q for a partial attention, and the pieces are accumulated with online softmax (the FlashAttention idea) — so communication can overlap with computation.

动图 · Ring:K/V 块沿环在卡间传递(P2P)Figure · Ring: K/V blocks passed between GPUs around the ring (P2P)

图 3 · KV 块绕环流动,每到一卡就算一次局部注意力。并行度不受 head 限制、可扩到超长序列;代价是通信量随切分增大,且 causal mask 下负载不均(后面的卡算得多)→ 用 Striped / ZigZag Attention 重排均衡Figure 3 · KV blocks circulate; each arrival triggers one local attention. Parallelism is not capped by head count and scales to very long sequences. The cost is that communication grows with the degree of sharding, and a causal mask makes the load uneven (later GPUs do more work) → rebalance with Striped / ZigZag Attention.

5 · USP / 混合(Ulysses × Ring,2D)5 · USP / hybrid (Ulysses × Ring, 2D)

Ulysses 通信省但受 head 数限制;Ring 不受 head 限制但通信量大。USP 把两者放进一个 2D mesh:组内用 Ulysses(All-to-All) 吃通信效率,组间用 Ring(P2P) 突破 head 上限。长上下文 + 大规模训练的常见选择。Ulysses is communication-cheap but capped by head count; Ring is uncapped but communication-heavy. USP puts both on a 2D mesh: Ulysses (All-to-All) within a group for communication efficiency, Ring (P2P) across groups to break the head-count ceiling. A common choice for long context at scale.

USP:两级并行,各取所长USP: two levels, each playing to its strength

组内 · Ulysses Within group · Ulysses

All-to-All,通信高效,受 head 数限制All-to-All, communication-efficient, capped by head count

组间 · Ring Across groups · Ring

P2P 环传递,突破 head 上限、扩超长序列P2P ring, breaks the head ceiling, scales to very long sequences

图 4 · USP = Ulysses(头维度高效) + Ring(序列维度可扩),代表实现:yunchang(USP)、LoongTrainFigure 4 · USP = Ulysses (efficient on the head dimension) + Ring (scalable on the sequence dimension). Reference implementations: yunchang (USP), LoongTrain.

6 · 四种方案优缺点对比6 · The four schemes compared

方案Scheme	通信原语Primitive	并行度上限Parallelism ceiling	优点Strengths	缺点Weaknesses
Megatron-SP	AllGather + ReduceScatter	= TP 度= TP degree	省非 TP 区激活显存;和 TP 无缝Saves activation memory outside the TP region; drops straight into TP	不解决长序列本身;绑定 TPDoes nothing for long sequences themselves; bound to TP
DeepSpeed-Ulysses	All-to-All	≤ head 数≤ head count	通信省、实现简单、与 seq 增长解耦Cheap communication, simple to implement, decoupled from sequence growth	并行度被 head 卡死;GQA 下更紧Parallelism hard-capped by heads; tighter under GQA
Ring / CP	P2P Send/Recv	不受 head 限制Not capped by head count	超长序列友好、通信可 overlapGood for very long sequences; communication can overlap	通信量随切分增大;causal 负载不均(需 ZigZag)Communication grows with sharding; causal load imbalance (needs ZigZag)
USP(混合)USP (hybrid)	All-to-All + P2P	突破 head 上限Breaks the head ceiling	兼顾通信效率与可扩性Balances communication efficiency against scalability	实现/调参更复杂(2D mesh)More complex to implement and tune (2D mesh)

怎么选:How to choose:

序列不算特别长、head 够分 → Ulysses(简单、通信省)。Sequence not especially long and enough heads to go around → Ulysses (simple, cheap).
序列非常长 / head 不够分 → Ring(CP),配 ZigZag 均衡。Very long sequence, or not enough heads → Ring (CP), with ZigZag for balance.
又长又要大并行 → USP 混合。Long and highly parallel → hybrid USP.
已经在用 TP、只想省激活显存 → 叠 Megatron-SP。Already on TP and only want the activation memory back → add Megatron-SP.
实战里这些还常和 DP / TP / PP / ZeRO 组成多维并行一起上。In practice these are usually stacked with DP / TP / PP / ZeRO into multi-dimensional parallelism.

7 · 主流大模型用什么通信原语7 · Which primitives the major models use

各并行维度对应的原语其实是固定的,记住这张映射表就能反推任何模型的通信特征:Each parallel dimension maps to a fixed primitive. Memorise this table and you can infer the communication profile of any model:

并行维度Parallel dimension	主要通信原语Primary primitive
数据并行 DP(朴素)Data parallel, DP (naive)	AllReduce
ZeRO / FSDP	ReduceScatter(梯度)+ AllGather(参数)ReduceScatter (gradients) + AllGather (parameters)
张量并行 TPTensor parallel, TP	AllReduce;开 SP 后 → AllGather + ReduceScatterAllReduce; with SP enabled → AllGather + ReduceScatter
序列并行 SPSequence parallel, SP	Ulysses → All-to-All;Ring/CP → P2P
流水线并行 PPPipeline parallel, PP	P2P Send/Recv
专家并行 EP(MoE)Expert parallel, EP (MoE)	All-to-All(token 派发 / 合并)All-to-All (token dispatch / combine)

以 DeepSeek(MoE + MLA)为例Case study: DeepSeek (MoE + MLA)

DeepSeek-V2/V3 是 MoE 架构,通信"主角"就是 All-to-All:每个 token 要被路由到分布在不同卡上的专家,再把结果收回 —— 这两步(dispatch / combine)就是两次 All-to-All,也是 MoE 训练/推理的主要通信瓶颈。DeepSeek-V2/V3 are MoE architectures, so the communication protagonist is All-to-All: every token has to be routed to experts spread across GPUs and the results gathered back. Those two steps — dispatch and combine — are two All-to-Alls, and the main communication bottleneck in MoE training and inference.

动图 · MoE 专家并行:token 经 All-to-All 派发到各卡专家Figure · MoE expert parallelism: tokens dispatched to experts via All-to-All

token(各卡)tokens (per GPU) 专家(各卡)experts (per GPU) G0 G1 G2 G3 E0 E1 E2 E3 图 5 · 每卡的 token 按路由结果发往对应专家所在的卡(dispatch),算完再 All-to-All 收回(combine)Figure 5 · Each GPU sends its tokens to whichever GPU holds the routed expert (dispatch); after the computation another All-to-All brings the results back (combine).

DeepEP:DeepSeek 自研开源的 EP 通信库,专门优化这两次 All-to-All —— 区分高吞吐 kernel(训练/prefill)与低延迟 kernel(decode),吃 NVLink(机内)+ RDMA/IB(机间),支持 FP8 通信。DeepEP: DeepSeek's own open-source EP communication library, built specifically around those two All-to-Alls — separate high-throughput kernels (training/prefill) and low-latency kernels (decode), using NVLink within a node and RDMA/IB across nodes, with FP8 communication support.
DualPipe:它的流水线并行方案(PP 用 P2P),把计算与通信(尤其 EP 的 All-to-All)充分 overlap,把通信开销藏到计算后面。DualPipe: their pipeline-parallel scheme (PP over P2P), which overlaps computation with communication — the EP All-to-Alls especially — hiding the communication behind compute.
其余维度照常:DP/ZeRO 用 AllGather + ReduceScatter,(若有)TP 用 AllReduce;MLA 主要省 KV cache,不改通信原语。The rest is conventional: DP/ZeRO use AllGather + ReduceScatter, and TP (where present) uses AllReduce. MLA mainly saves KV cache and does not change the primitives.

注意:"ds" 若指 DeepSpeed(框架) 而非 DeepSeek(模型):ZeRO 用 ReduceScatter + AllGather,DeepSpeed-Ulysses 用 All-to-All。A note on naming: if "ds" means DeepSpeed (the framework) rather than DeepSeek (the model), then ZeRO uses ReduceScatter + AllGather and DeepSpeed-Ulysses uses All-to-All.

8 · 一句话总结8 · In two sentences

两句话拎清Two sentences to keep

① SP 方案:省 TP 激活显存用 Megatron-SP(AllGather+ReduceScatter);长序列三选一 —— Ulysses(All-to-All,受 head 限)、Ring/CP(P2P 环,不受 head 限、超长友好但需 ZigZag 均衡)、USP(2D 混合,取长补短)。
② 大模型通信原语:DP→AllReduce、ZeRO→ReduceScatter+AllGather、TP→AllReduce、PP→P2P、SP→All-to-All/P2P、MoE→All-to-All;DeepSeek 这类 MoE 的主角是 All-to-All,用 DeepEP 优化、DualPipe overlap。① SP schemes: to save TP activation memory use Megatron-SP (AllGather+ReduceScatter); for long sequences pick one of three — Ulysses (All-to-All, head-capped), Ring/CP (P2P ring, uncapped and good for very long context but needs ZigZag balancing), or USP (2D hybrid, each covering the other's weakness).
② Communication primitives: DP→AllReduce, ZeRO→ReduceScatter+AllGather, TP→AllReduce, PP→P2P, SP→All-to-All/P2P, MoE→All-to-All; for MoE models like DeepSeek the All-to-All is the bottleneck worth optimising.

图 6 · 记住"哪个并行用哪个原语",就能反推任何大模型的通信特征Figure 6 · Remember which parallelism uses which primitive and you can infer any model's communication profile

参考:Megatron-LM(Sequence Parallelism)、DeepSpeed-Ulysses 论文、Ring Attention / Striped Attention、Megatron-Core Context Parallelism、USP(yunchang)/ LoongTrain、DeepSeek-V3 技术报告与 DeepEP / DualPipe 开源仓库、NCCL/HCCL 集合通信文档。细节可能随版本演进。References: Megatron-LM (Sequence Parallelism), the DeepSpeed-Ulysses paper, Ring Attention / Striped Attention, Megatron-Core Context Parallelism, USP (yunchang) / LoongTrain, the DeepSeek-V3 technical report and DeepEP.