SXEAI(Sextic Equation AI)

Abstract

---license: mit--- # SXEAI: Reverse-Engineering the Mathematics Learned by a Neural Sextic Solver ## Abstract Can a neural network learn not only to solve an algebraic problem, but also to develop an internal mathematical representation of the problem that we can reverse-engineer? This is the main question behind **SXEAI**, an experimental project focused on neural networks that recover the six roots of a general polynomial of degree six from its coefficients. The objective is deliberately different from a conventional machine-learning benchmark. We are not primarily interested in obtaining the smallest possible root prediction error. Instead, we want to understand **what the network computes internally**. Across a sequence of experiments, SXEAI repeatedly developed hidden representations that could be approximated by compact combinations of symmetric polynomial quantities, power sums, centered moments, depressed-polynomial coefficients, and related algebraic features. Some of these internal coordinates could be discovered blindly, reproduced on unseen data, and even replaced causally by their symbolic approximations without substantially changing downstream network behavior. The resulting picture is not yet a discovered closed-form solution of the general sextic equation. It is something more precise and, in our view, more interesting: evidence that a neural algebraic solver can spontaneously organize its hidden state into a structured mathematical coordinate system that can be partially reconstructed and experimentally manipulated. --- # 1. The question behind SXEAI A standard neural network experiment asks a simple question: > Can the network learn the mapping from inputs to outputs? For a sextic polynomial, $$P(z)=z^6+p_5z^5+p_4z^4+p_3z^3+p_2z^2+p_1z+p_0,$$ the conventional task would be: $$(p_5,p_4,p_3,p_2,p_1,p_0)\longrightarrow(r_1,\ldots,r_6),$$ where $$P(z)=\prod_{i=1}^{6}(z-r_i).$$ One can then measure MAE, RMSE, maximum root error, or polynomial reconstruction error. SXEAI asks a different question: $$\boxed{\text{What mathematical representation does the network build internally?}}$$ The network may not need to directly represent each root at every stage. It could instead construct intermediate quantities such as $$-\frac{p_5}{6},$$ power sums $$s_k=\sum_i r_i^k,$$ centered moments $$q_k=\frac16\sum_i(r_i-\mu)^k,$$ or more complicated combinations such as $$s_2s_3,\qquad s_2s_4,\qquad q_3q_4.$$ The central idea of the project is therefore to treat the network itself as a mathematical object that can be reverse-engineered. --- # 2. Why the sextic problem? The sextic is an interesting test case because it is already beyond the regime where one should expect a simple universal elementary formula for the roots. For low-degree equations, classical symbolic methods provide familiar closed forms. At degree six, the algebraic structure becomes substantially richer. This makes the sextic a useful environment for asking: > Does a neural solver rediscover recognizable algebraic coordinates, or does it develop a completely different internal representation? We do not assume in advance that the answer must involve radicals. That distinction became increasingly important as the project progressed. --- # 3. The model SXEAI uses a custom layer called `SexticLayer`. A layer first computes $$z=Wx+b,$$ then applies a trainable sixth-degree polynomial $$Az^6+Bz^5+Cz^4+Dz^3+Ez^2+Fz+G,$$ followed by normalization and GELU. The polynomial activation is initialized close to a quadratic form, so the network starts from a relatively simple nonlinear regime but can learn higher-order behavior. The main complex model used in the later experiments has approximately 1.2 million parameters and consists of 24 such layers with hidden width 223. The final network maps a 12-dimensional real representation of the polynomial coefficients to 12 real values representing six complex roots: $$[\Re r_1,\Im r_1,\ldots,\Re r_6,\Im r_6].$$ This architecture is intentionally unusual. Rather than hiding all nonlinearity inside a standard ReLU or GELU network, we give the model explicit trainable polynomial operations and then ask what mathematical structure emerges. --- # 4. Data generation The polynomial is generated from its roots. For six roots $$r_1,\ldots,r_6,$$ we construct $$P(z)=\prod_{i=1}^{6}(z-r_i).$$ The network does not receive the roots. It receives only the resulting coefficients. For the complex experiments, roots are represented by real and imaginary parts: $$r_i=a_i+ib_i.$$ The root generation, normalization, coefficient construction, ordering and scaling are kept fixed across the main comparisons. The training protocol used in the main experiments includes: * 350,000 training samples;* batch size 768;* 125 epochs;* AdamW;* learning rate \(3\times10^{-4}\);* weight decay \(10^{-4}\);* gradient clipping at 1.0;* SmoothL1 loss. The dataset mixes real, natural/integer-like and fully complex root configurations. --- # 5. The first discovery: the network finds the center One of the earliest and most persistent observations was the appearance of $$\boxed{\mu=-\frac{p_5}{6}}.$$ Since $$p_5=-\sum_i r_i,$$ we have $$\mu=\frac16\sum_i r_i.$$ In other words, the first layers repeatedly discovered the **mean of the roots**. This was not a one-off correlation. Across different architectures and training runs, early hidden units consistently exhibited strong alignment with this quantity. That suggested an important possibility: > the network may first transform the problem into a coordinate system centered around the roots before attempting to recover their detailed structure. --- # 6. From coefficients to symmetric quantities The next step was to search hidden activations against a library of algebraically meaningful quantities. Among the most important were the power sums $$s_k=\sum_i r_i^k,$$ together with centered moments $$q_k=\frac16\sum_i(r_i-\mu)^k$$ and the coefficients of the depressed sextic obtained after translating the variable by the root mean. Newton identities provide the connection between these representations. For example, $$s_1=\sum_i r_i=-p_5,$$ and $$s_2,\;s_3,\ldots$$ can be expressed recursively in terms of the polynomial coefficients. This means the network has access, through ordinary algebraic transformations, to several equivalent coordinate systems describing the same root configuration. The symbolic experiments suggested that it was not merely using raw coefficient coordinates. It repeatedly rediscovered combinations of the more natural symmetric quantities. --- # 7. Deep layers developed richer combinations In deeper networks we began to find hidden units whose activations could be approximated by combinations such as $$a s_3+b s_5+c\,s_2s_3,$$ or $$a s_6+b\,s_2s_4+c\,q_4.$$ Examples from earlier high-performing real-root models included expressions such as $$1.7637s_3+0.2457s_5-1.2059s_2s_3$$ with \(R^2\) around 0.80, and $$-0.7720s_1-0.7597s_3+0.6254s_2s_3$$ with \(R^2\) around 0.89. The exact coefficients are not the scientific result by themselves. The important observation is the repeated vocabulary: $$s_1,\quad s_2,\quad s_3,\quad s_5,\quad s_6,\quad s_2s_3,\quad s_2s_4,\ldots$$ These objects have a clear mathematical interpretation as symmetric functions of the roots. --- # 8. The architecture did not scale monotonically One of the strongest negative results appeared when we increased network depth. Larger models did not automatically produce better numerical or symbolic behavior. Some 32-layer configurations entered a collapsed regime in which the output became almost constant across very different polynomial inputs. A representative failure had approximately $$\mathrm{MAE}\approx12.5,$$ compared with values near 1 or below 1 for successful real-root runs. This happened even when the parameter budget was large. Later, a particularly clean comparison used approximately the same parameter count: $$24L/600K$$ versus $$32L/600K.$$ The 24-layer model worked normally, while the 32-layer model collapsed after only a few epochs. This is an important result because it shows: $$\boxed{\text{more depth does not automatically produce more algebraic reasoning}}$$ and, in this architecture, depth interacts strongly with optimization stability. --- # 9. A second important observation: optimization is part of the phenomenon Some models trained well for many epochs and then suddenly deteriorated. For example, a 16-layer model reached a very low training loss before a later catastrophic transition. Other models collapsed much earlier. This means that SXEAI is not just about the final trained state. The **trajectory of learning itself** contains information. A model can pass through a phase where its internal representation is highly structured, and later destroy that representation through optimization instability. That led to one of the most interesting changes in the project: > Instead of analysing only the final checkpoint, we began analysing the hidden representation as a function of training epoch. --- # 10. A stricter symbolic protocol Our early symbolic experiments had several weaknesses. First, some features were algebraically dependent. For example, $$p_5=-6\mu.$$ Therefore $$p_5^2=36\mu^2.$$ Treating these as independent regression coordinates can create misleading sparse formulas. Second, fitting and evaluating on the same probe points can overestimate symbolic quality. Third, a very high \(R^2\) can sometimes be produced by a nearly constant hidden unit. We therefore moved toward a stricter procedure: * separate symbolic fit and holdout sets;* larger feature libraries;* numerical rank reduction;* removal of obvious aliases;* sparse formulas with one to three main terms;* explicit checks for hidden activation variance and non-triviality. This changed the inter