J. Melton

Efficient Mechanistic Circuit Analysis of Open-Weight LLMs on Apple Silicon

Replication and extension of the GPT-2 Small IOI circuit analysis on Llama-3.2-3B and Pythia-1.4B using an MLX activation-patching harness on a single Apple Silicon machine. Circuits are faithful on the standard single-template dataset (F=0.663 and 0.854) but collapse on a 15-tem...

TopK Sparse Autoencoders Across Three Model Architectures: Dictionary Collapse, Dense-Feature Degeneracy, and the Limits of Activation-Pattern Feature Matching

TopK SAEs trained at matched sparsity and dictionary size on Llama-3.2-3B, Mistral-7B-v0.3 and Qwen2.5-3B. Reports abrupt dictionary collapse with partial recovery, dense-feature degeneracy invalidating frequency-based feature selection, and a null cross-architecture matching res...

Distributed Mechanistic Interpretability at Scale: Activation Streaming, Split-Layer Inference, and Distributed Sparse Autoencoder Training

ActStream: a two-node system that splits inference and analysis, streams intermediate activations over a binary wire protocol, and trains sparse autoencoders without staging activations to disk. Thunderbolt measured at 2.150 GB/s, 86% of negotiated rate and 26x the LAN path, with...