A Framework for Using Metadata to Combine Data Sets in Official Statistics

Abstract

There are increasing numbers of sources available for the production of official statistics. When one is interested in making new output or redesigning the input that is used for existing output, it might not always be clear if and how the intended output can be produced from the variety of input data sets available. The current paper proposes a metadata-based framework with a search algorithm that finds out if a processing path is possible from input data sets to intended output. The framework uses the conceptual metadata of the data set as input, that is, the unit types, the identifier variables, the measurement variables and the reference periods. Our contribution to existing literature is threefold. We provide a set of basic operators, each representing a small, clearly defined step in the process of combining data sets; we can accommodate tailor-made operators (models) to cover a wide range of data configurations, and we provide an implementation in Python code to automatically answer if and how the intended output can be produced from a set of available input data sets. The framework is evaluated on a case study. We provide publicly available code and different examples on Git to facilitate the use and further development of our framework.