Ael-504M: A Sub-Billion Parameter Hybrid Mamba-2 and Delta-Attention Any-to-Any Multimodal Architecture

Abstract

Ael-504M is an original sub-billion parameter multimodal foundation model engineered for high-throughput on-device reasoning and cross-sensory synthesis. Key Architectural Innovations: Dual-Brain Sequence Modeling: Interleaves Mamba-2 State Space Duality (SSD) for continuous O(L) linear recurrence with Rotary Delta-Attention (RDA) for O(L²) geometric focal precision. Centroid-Routed SwiGLU MoE: 8 specialized expert pathways with cosine centroid gating activating only ~182M parameters per token for ultra-efficient execution. Any-to-Any Tri-Modal Fusion: Native cross-attention fusion projecting Vision (224x224 Conv2D patches), Audio (80-Mel spectrograms), and Text (50,304 tokens) into a shared 1024-dimensional latent manifold. On-Device Footprint: Operates within a compact ~1.01 GB VRAM footprint, designed for edge robotics, laptops, and mobile hardware. Author: Prannessh, K. V. A. License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) Repository: https://huggingface.co/Prannesshkva/Ael-504M