The sign of the selected candidate's P&L conflates the level of the test period with the quality of selection

Abstract

The standard walk-forward step reports the sign of the out-of-sample (OOS) P&L of a single candidate: the one the selection rule ranked first in-sample (IS). That sign conflates two facts: what kind of period the test period was for the whole candidate set, and whether the selected candidates' advantage survived. The familiar labels (non-robust, overfit, bad period) do not answer this question. To make this visible, I build a simulation testbed in which the level of the period and the ordering among candidates are set in advance. On it one can see when the sign is wrong and the position of the selected top slice is not. Alongside it sits a comparison against candidates with the same number of trades: it removes the part of the level that passes through trade count, but it does not replace the P&L. Six windows of one trading system and six windows of another algorithm show the same reading on real history. How often the sign is wrong on other people's data is not estimated by these windows.