Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Abstract

Long‑context large language models suffer from two coupled information dilution issues: flattened softmax weights and compressed KV representations on the sequence dimension, and gradually obscured early‑layer signals along the depth dimension. Existing solutions rely on hard token dropping or static layer block partitioning, which either permanently discard information or fail to adapt to input complexity. In this work, we propose a unified soft‑adaptive aggregation framework that treats sequence‑side and depth‑side dilution as a dual problem. On the sequence axis, we introduce Soft‑ChunkAttn, which partitions history into semantic chunks via differentiable Gumbel‑Sigmoid boundary prediction, adopts attention‑weighted chunk summarization, and performs full softmax aggregation over all chunks without hard top‑k dropping. An ALiBi‑style relative position scheme is designed for chunk summaries to improve extrapolation. On the depth axis, we present Virtual Dynamic Block‑AttnRes, which generates input‑adaptive virtual block summaries via inter‑layer similarity‑based merging weights, with a dedicated regularization term to stabilize training and prevent granularity collapse. Critically, the physical computation graph remains fully unchanged, and vectorized segment operators are adopted for efficient GPU deployment, preserving compatibility with distributed training pipelines. We present the complete methodological design and theoretical analysis of the proposed framework. Full empirical validation, including ablation studies and large‑scale pre‑training results, will be supplemented in subsequent versions. Contact for discussion and collaboration:m3e1-a@outlook.com This is a single‑authored pre‑print work. The work may contain imperfections and unaddressed issues. Further empirical evaluations will be added in future revisions. Constructive feedback, critical comments and potential collaborations are highly welcome.