SATIN: a sparse attention application-specific instruction-set processor with custom mixed-precision extensions for energy-efficient edge inference of transformer large language models

Abstract

Edge inference of Transformer large language models is constrained by the quadratic cost of self-attention and by the memory traffic of the key-value cache. Existing accelerators treat sparsity and mixed precision as separate fixed-function mechanisms and provide no accuracy guarantee for an individual inference. This paper proposes a sparse attention application-specific instruction-set processor (SATIN). SATIN treats pruning as the zero-bit extreme of a bit-width continuum and thereby unifies sparsity and mixed precision as bit-plane precision. Partial sums from most-significant-bit-first serial computation supply an importance signal at low additional cost, together with a monotonically tightening error bound. One signal drives three coupled mechanisms: token sparsity on the query and key side, graded precision on the value side in proportion to the softmax weights, and importance-adaptive fetch depth for the key-value cache. A custom bit-plane instruction-set extension and a global accuracy budget allow every inference to carry a verifiable accuracy certificate. On public language and vision benchmarks the accuracy loss at an accuracy budget of one percent stays near half a percentage point. SATIN attains an effective energy efficiency of about 43.5 tera-operations per second per watt and a speedup of about 2.33 times over full precision, and reduces long-context key-value memory traffic to about one third of the dense baseline while remaining programmable.