Samuel Mausberg

Exact Sampling from Neural Networks: Depth, Attention, and Decoding

One random output can require much less work than numerical evaluation. We study exact neural sampling, counting input probes, random bits, arithmetic, and every precision refinement. For dense tanh networks with absolute row sums at most one, weight preprocessing permits polylog...

Near-Linear Attention in Three Dimensions

Given n query vectors, n key vectors, and scalar values, softmax attention returns one exponentially weighted average per query. We give an n 2O(√log n)-time randomised algorithm in three dimensions on a RAM with O(log n)-bit words. Coordinates are logarithmic-bit rationals bound...