Developing a summative and integrated framework to assess living lab interventions: Insights and reflections from a multi-site case study

Abstract

The impact assessment of a Living Lab represents a crucial research challenge given the complexity and diversity of stakeholders and characteristics of Living Labs (e.g., context, scales, and goals). While the literature emphasizes formative approaches focused on performance using qualitative methods, summative assessments with quantitative methods remain underdeveloped, despite their importance in evaluating the final outcomes and demonstrating the effectiveness and accountability of Living Lab interventions. So far, there is limited documented experience and guidance on precise processes, tools, and methods on how to construct a summative and integrated impact assessment of Living Labs that is scientifically robust and operational., This study addresses this gap by demonstrating and detailing a novel and process-oriented framework for developing a summative and integrated impact assessment that is comprehensive, operational, and contextually and broadly useful in comparing the impacts of interventions within a Living Lab and across Living Labs. To achieve this, we draw on a case study of a Living Lab project in Switzerland that integrates multiple existing urban Living Labs. We demonstrate a four-stage process for developing a summative and integrated impact assessment, including a co-design approach as a fundamental component throughout. The process consists of (i) co-specifying the constructs of a summative and integrated impact assessment; (ii) co-creating an intervention-specific Theory of Change (ToC); (iii) translating the ToC into a logical framework with inputs, outputs, outcomes, and impacts; and (iv) defining measurable indicators corresponding to the identified constructs. Finally, we report challenges, lessons learned, and recommendations regarding the experience of co-developing a summative impact assessment. These can support other programs in their impact assessment endeavours.