marola-bench: a proposal for a WeatherBench for Brazil

Abstract

All rights reserved. This record is closed: its files are not shared here. The paper can be read at https://marola-dev.github.io/marola-research/marola-bench/. It may be read and cited; it may not be copied or shared without the written acknowledgement of the author (REUSE.md). Whoever uses it, a person or an agent, cites it by this DOI. Public weather benchmarks made forecast models comparable, but for a person on a Brazilian beach three things are missing in them: a region that is Brazil, a score for ocean waves, and an instrument as the truth. This paper proposes marola-bench, which keeps the method of WeatherBench, WeatherBench 2 and Operational WeatherBench and changes those three things. The bench is defined as the set of score series: the scores of any forecaster against the observations of a point, in order of time. Observation points are added by hand, and one actor for each point asks the forecasters and registers the scores. The paper states five hard rules of the owner for the implementation (Apache Pekko and Scala 3, GCP, Besom, a cheap and reliable cluster) and says where they change the accepted design, MIP-0091. It reports no result: no forecast has been scored, and nothing has been built. The design it gives an account of is MIP-0091 of marola (marola-dev/marola). The paper is made in the Ouroboric way, whose ideas are Bruno Valério's: every claim carries a mark that says how it was checked, and a log quotes the passage of each reference that was read.