Analysis of a Large-Scale Citizen Science Study on the Public Collaborative Reassembly of an Archaeological Stone Artifact

Abstract

Archaeologists often face the complex and time-consuming task of reassembling fragments from broken artifacts. Fragment reassembly is traditionally done manually by domain experts. Automatic algorithms have advanced but are typically tailored to specific fragment types, often assuming indicative features such as surface paintings or relief decorations. Faced with incomplete and weathered fragments, fully automatic reassembly algorithms can rarely guarantee correctness. These observations indicate that neither a purely manual nor a fully automatic solution is optimal for such tasks. We conducted a user study using Open Reassembly, a web-based platform that makes fragments of an incomplete, ancient stone artifact accessible to the general public and offers means for their virtual reassembly in the form of a collaborative puzzle game. Besides manual reassembly of digitized fragments, the framework offers optional computer-aided tools that facilitate the search for matching fragments and their alignment, and displays a calculated geometric match-quality score for each joined fragment pair. Participants can also assess matches qualitatively through a built-in voting system, thereby sharing their opinion on the correctness of fragment connections. We pose three research questions: (i) how well can users find correct matches between stones, (ii) how does collaboration among users shape the reassembly outcome, and (iii) how well can users assess the correctness of fragment connections. We answer these questions by analyzing the interactions of more than 5000 users. A second controlled study with 11 participants allows us to evaluate the framework's usability. Our results show that a small number of highly active users disproportionately shape the result, that additional automated assistance does not by itself improve outcomes, that user voting is a reliable proxy for join correctness when no ground truth is available, and thus that crowd-sourcing frameworks are suitable for reassembly tasks.