Xiaofeng Qi

ATAC: Anchor-tail aware context parallelism for LLM training

Large language model training commonly relies on multidimensional parallelism, including data, tensor, pipeline, and context parallelism, to support long-context and large-scale workloads. However, real pretraining corpora consist of highly heterogeneous variable-length samples,...