Modern LLM pretraining leverages pipeline parallelism to enhance throughput. Recent research highlights asynchronous pipeline parallelism, especially PipeDream-2BW, as an effective solution that maintains consistent gradient delays. This approach avoids wasted GPU cycles during pipeline bubbles. Read original