Allen Institute for Artificial Intelligence

Retrofitting language models to operate over bytes

Abstract Recent advances in artificial intelligence (AI) have largely been driven by large language models, deep neural networks that operate over discrete units called tokens. To represent text, most large language models use words or word fragments as the tokens, known as subwo...

A General Theory of Neural Fields: Width, Attention, Covariance Processes, Renormalization, Gauge Quotients, and Quantum Reconstruction

We develop a mathematical framework for neural field laws that separates width, head count, sequence length, depth, training, coarse-graining, parameter quotients, and quantum reconstruction. Under a declared conditional Gaussian representation, connected field correlators are de...

Foundations of Formal Market Microstructure Large in U.S. Equities: Machine-Checked Allocation, Composition, Necessary State, and Regulatory Traceability

This paper develops executable models of price--time priority and parity allocation in U.S. equities to establish when execution consistency holds and which state must be retained. Splitting an order can change who receives shares. In a parity wheel, resetting the rotation betwee...

The Economics of Artificial Intelligence: Scaling, Verification, Assignment, Capital, Growth, and Value

This paper develops the economics of artificial intelligence as a single connected structure, from the physics of the production function to the aggregate growth constraint and the valuation of the firms building and adopting it. Part I derives the cost of capability from scaling...

A New Software Paradigm: Trained Systems, Large Language Models, Agents, Verification Harnesses, and Proof-Carrying Software

Software is no longer only written; increasingly it is trained. The artifact this produces is not syntax given meaning by an operational semantics but a probability law coupled to an environment---and none of the classical guarantees survive the change unmodified. This paper supp...

A New Software Paradigm: Trained Systems, Large Language Models, Agents, Verification Harnesses, and Proof-Carrying Software

Software is no longer only written; increasingly it is trained. The artifact this produces is not syntax given meaning by an operational semantics but a probability law coupled to an environment---and none of the classical guarantees survive the change unmodified. This paper supp...

The Reflexive Path-Law Framework: Unpredictability, Identification, and Market Equilibrium

Return unpredictability is an observational property relative to a filtration, horizon, target, loss, and strategy class; it does not by itself establish correct valuation, no-arbitrage, net non-exploitability, causal exogeneity, or dynamic stability. This paper develops the Refl...

The Reflexive Path-Law Framework: Unpredictability, Identification, and Market Equilibrium

Return unpredictability is an observational property relative to a filtration, horizon, target, loss, and strategy class; it does not by itself establish correct valuation, no-arbitrage, net non-exploitability, causal exogeneity, or dynamic stability. This paper develops the Refl...

The Mathematics of Factors: Risk, Alpha, Detection, and Capacity

Quantitative finance uses "factor" for two distinct objects: a risk direction along which returns covary and an alpha direction along which returns are predictable. We assign each direction gauge-invariant common-covariance strength and predictive content, separating covariance r...

Density Modeling of Sequences: A Path-First Theory of Autoregression, Masked Diffusion, Adaptive Insertion, and the Price of Order

We develop a path-first theory of discrete sequence generation. A generator first defines a probability law on chronology-augmented construction paths; the terminal sequence law is the pushforward of that path law. The full geometric signature faithfully encodes every finite acti...

Density Modeling of Sequences: A Path-First Theory of Autoregression, Masked Diffusion, Adaptive Insertion, and the Price of Order

We develop a path-first theory of discrete sequence generation. A generator first defines a probability law on chronology-augmented construction paths; the terminal sequence law is the pushforward of that path law. The full geometric signature faithfully encodes every finite acti...

Structure Through Five Geometries: From Gauge Quotients to Transformer J-Space

This paper organizes structural questions in artificial intelligence and finance through five complementary geometries. Gauge geometry studies equivalent internal descriptions. Orbit and quotient geometry studies observational identification. Information geometry measures local s...