Minzhe Liu

Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Long‑context large language models suffer from two coupled information‑dilution issues: flattened softmax weights and compressed KV representations along the sequence dimension, as well as progressively obscured early‑layer signals across the depth dimension. Existing approaches...

Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Long‑context large language models suffer from two coupled information dilution issues: flattened softmax weights and compressed KV representations on the sequence dimension, and gradually obscured early‑layer signals along the depth dimension. Existing solutions rely on hard tok...