Minzhe Liu

Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Long‑context large language models suffer from two coupled information dilution issues: flattened softmax weights and compressed KV representations on the sequence dimension, and gradually obscured early‑layer signals along the depth dimension. Existing solutions rely on hard tok...

Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Long‑context large language models suffer from two coupled information dilution issues: flattened softmax weights and compressed KV representations on the sequence dimension, and gradually obscured early‑layer signals along the depth dimension. Existing solutions rely on hard tok...