[ad_1]
\n<\/p>\n
Pipeline parallelism splits a model \u201cvertically\u201d by layer. It\u2019s also possible to \u201chorizontally\u201d split certain operations within a layer, which is usually called\u00a0Tensor Parallel<\/em>\u00a0training. For many modern models (such as the\u00a0Transformer<\/a>), the computation bottleneck is multiplying an activation batch matrix with a large weight matrix.\u00a0Matrix multiplication<\/a>\u00a0can be thought of as dot products between pairs of rows and columns; it\u2019s possible to compute independent dot products on different GPUs, or to compute parts of each dot product on different GPUs and sum up the results. With either strategy, we can slice the weight matrix into even-sized \u201cshards\u201d, host each shard on a different GPU, and use that shard to compute the relevant part of the overall matrix product before later communicating to combine the\u00a0results.<\/p>\n One example is\u00a0Megatron-LM<\/a>, which parallelizes matrix multiplications within the Transformer\u2019s self-attention and MLP layers.\u00a0PTD-P<\/a>\u00a0uses tensor, data, and pipeline parallelism; its pipeline schedule assigns multiple non-consecutive layers to each device, reducing bubble overhead at the cost of more network\u00a0communication.<\/p>\n Sometimes the input to the network can be parallelized across a dimension with a high degree of parallel computation relative to cross-communication.\u00a0Sequence parallelism<\/a>\u00a0is one such idea, where an input sequence is split across time into multiple sub-examples, proportionally decreasing peak memory consumption by allowing the computation to proceed with more granularly-sized\u00a0examples.
<\/p>\n<\/div>\n<\/div>\n