Designing and Evaluating Human–AI Interaction Service Systems: An Industrial Case Study

Abstract

This study advances the design and evaluation of Human–AI Interaction Service Systems (HAI-SS) through a design science research approach in collaboration with a global high-tech manufacturer. Two artifacts were developed. The first, a multi-agent prototype, demonstrates how hybrid intelligence can be operationalized to support diagnostic work, standardize knowledge reuse, and address service-quality objectives across support, customization, and commissioning. The second, a HAI-SS Assessment Model, consolidates 51 metrics and 40 evaluation methods into 8 clusters, linking goals, metrics, and methods through conditional rules to enable context-sensitive evaluation. Together, these artifacts contribute methodological rigor, empirical grounding, and actionable tools for both research and practice. By introducing HAI-SS as a socio-technical category and providing a replicable evaluation framework, the study advances research in service science and information systems and directs the responsible incorporation of AI into complex service systems.