I recently saw a presentation by Dick Kasperowski and Olof Karsvall about research they’d done on The Detective Section, a crowdsourcing project at the Swedish National Archives. Their findings contradicted some of the conventional wisdom that volunteers introduce bias into results when they bring their own subject-matter expertise to citizen science tasks, and that made me perk up my ears. Dick, Olof, and their collaborator Karl-Magnus Johansson were kind enough to share their research with us, and I’d like to share some of the things that struck me about their paper.
The Swedish National Archives asked volunteers to transcribe police reports from 1865-1903 recording petty crimes in Gothenburg. Their work was part of an effort to train an AI to transcribe such records, creating a Handwritten Text Recognition (HTR) model in Transkribus. To this end, they needed initial transcriptions of 400 images to begin training the model, then a second phase correcting AI-produced transcripts to fine-tune the model. Five volunteers participated in the first phase, joined by eighteen more in the second phase.
Motivation
The researchers interviewed eight participants, focusing on what the experience was like for volunteers, and what they got out of the process. Overwhelmingly, volunteers valued the immersive nature of working with these documents:
When you get close to the people you read about in the material, you sometimes feel that you know specific bicycle thieves.
The larger project goals of making the archives more accessible or working with cutting-edge technology were far less important to participants. Learning about life in 19th-century Gothenburg was rewarding outside of the project itself – one volunteer tells her friends about cases in the police reports during their walks, connecting those stories to the locations they pass.
Outreach
Volunteers joined the project because they trusted the National Archives to make the experience meaningful and worthwhile. This trust was based on their previous experience with lectures and participatory programs run by the archives over previous years. I was astonished to learn that the participants all joined because of invitations in the Archives newsletters, rather than the social media posts which reached a much wider audience. This speaks to the importance of long-term public programming as a basis from which to launch successful crowdsourcing projects.
Quality
One of the things participants talked about was the value of taking time during the process, and how that improves the quality of the results.
At the first glance, I see 6–8 errors per page. Then I switch to the level of meaning and it is at this instance that I understand the text for the first time. Earlier, it was just word for word to get the correction of the transcription right, not on the level of meaning. There is no flow when you concentrate on the first correction. [...] When I switch to the level of meaning I find even more mistakes. Then I let the text rest for about two weeks before returning to it, controlling it letter by letter, finding at least one to two additional mistakes per page.
Research in sources outside of the project texts was crucial to quality, as volunteers consulted old maps, census records, and newspapers to identify places and people written in difficult handwriting. Volunteers shared these resources with each other, and even created new resources, like “a glossary of different textiles, as the most reported crime was the theft of clothes and other textiles.”
The researchers conclude that, rather than creating bias, the local knowledge developed by participants and the emotional connection to the material was crucial to the quality of transcription and HTR correction:
A recurrent narrative is the participants’ accounts of specific local historical knowledge as an important asset for the quality of the correction of transcriptions. The more you develop your knowledge about the local history, in fact establish a personal and emotional relationship to the police officers and scribes, as well as delinquents, the better you can accomplish the task of training the HTR model.
Personally, I love hearing that the meaningful, immersive aspect of transcription also produces higher-quality results, since we designed FromThePage with those ideals in mind. Transcribers can make multiple revisions of the same page over time; collaborate with each other on the page, in the comments or on the forum; and work sequentially through material.
The full paper is online here: Kasperowski D., Johansson K.-M. and Karsvall O. (2024) “Temporalities and Values in an Epistemic Culture: Citizen Humanities, Local Knowledge, and AI-supported Transcription of Archives”, Archives & Manuscripts, 510, p. e10937. doi: 10.37683/asa.v51.10937.