Ael-504M: A Sub-Billion Parameter Hybrid Mamba-2 and Delta-Attention Any-to-Any Multimodal Architecture

Abstract

Ael-504M is a sub-billion parameter (504M) multimodal foundation model designed for on-device reasoning and cross-sensory synthesis. The architecture features a 24-layer hybrid backbone integrating Mamba-2 State Space Duality (SSD) with Rotary Delta-Attention (RDA), backed by 8-Expert Centroid-Routed SwiGLU Mixture-of-Experts (MoE) and an Any-to-Any Tri-Modal Fusion Hub (Vision, Audio, Text). Operating within a ~1.01 GB VRAM footprint, Ael-504M enables lightweight edge AI deployment while preserving dense multimodal expressive capacity.