A Proper Subclass of the Separable Permutations Whose Generating Function Lies Outside Q(z, √(1−4z), √(1−6z+5z²))

Abstract

Question 5 of R. Brignall's abstract (joint work with M. Bóna, C. Defant, M. Opler and V. Vatter) in Oberwolfach Report 6/2024, p. 292, asks whether every proper subclass of the separable permutations Av(2413, 3142) has a generating function in K = Q(z, √(1−4z), √(1−6z+5z²)). The answer is no. The class C = Av(2341, 2413, 3142) is a proper subclass of the separable permutations which contains Av(231). Its generating function f = Σ_{n≥0} |C_n| z^n satisfies z(1−z+z²)f² − (1−2z+2z²)f + (1−z)² = 0, so that f = (1−2z+2z² − √D)/(2z(1−z+z²)) with D = 1−8z+20z²−24z³+16z⁴−4z⁵. This enumeration is due to Callan, Mansour and Shattuck (Discrete Math. Theor. Comput. Sci. 19:1 (2017), #6, Theorem 11), and the same generating function is in the PermPAL database. The polynomial D is irreducible over Q and coprime to (1−4z)(1−z)(1−5z), whereas a nonzero element of Q(z) is a square in K only if it lies in the square class of 1, 1−4z, (1−z)(1−5z) or (1−4z)(1−z)(1−5z); hence √D ∉ K and f ∉ K. The contribution of this note is the observation that this class answers Question 5 negatively, a self-contained proof of its enumeration by the substitution decomposition, and an exact certificate that its generating function is not in K. This is an unrefereed note. Unrefereed preprint released for independent mathematical scrutiny. Publication on Zenodo does not constitute peer review. AI-assisted tools supported research, computation, proof development, and manuscript preparation. The author remains responsible for all claims and the final text. Corpus identifier: OWR-14298158-012 (R. Brignall et al., Question 5, Oberwolfach Report 6/2024, p. 292).