Human-AI Collaboration for Qualitative Analysis

Large datasets are typically the domain of quantitative scholars, who use computational tools to derive numerical insights. Qualitative researchers, who build deep and contextualized understandings of text such as interviews, open-ended surveys, and social media posts, have had far fewer tools to work with at scale. This project asked how computation and AI might support qualitative scholars in developing rich, interpretive understandings of large corpora, without reducing qualitative analysis to automation.

The project was funded by the National Science Foundation (CHS: Medium, Scaling Qualitative Inductive Analysis through Computational Methods, Award 1764089, 2018 to 2022) and was a collaboration between the Identity Lab, Danielle Albers Szafir’s visualization lab, and Casey Fiesler. Its central move was to invert the usual “human-in-the-loop” model, in which people verify what a computational system produces. Instead, the work starts from the insights of human analysts and asks how computation can amplify them.

What we did

Studying the work of qualitative researchers. We began by studying how qualitative researchers actually work. Interviews with qualitative scholars described their diverse methodological practices and the messiness and uncertainty of inductive analysis. Scholars wanted full agency over the process, and saw serendipity as something AI should not interfere with. This work framed the question as one of task delegability: which parts of qualitative analysis could people reasonably hand to a machine, and which not.

Time and perspective. A follow-up study with 15 CSCW and HCI qualitative researchers showed that the stage of analysis matters. Scholars were open to AI assistance in data exploration and coding, as long as it assisted rather than automated their analytic work, and incorporating AI changes some analytic practices and trades off scale, abstraction, and perspective.

Building tools. With visualization and machine learning collaborators, we designed and tested systems for inductive analysis. QualVis, presented at the Machine Learning from User Interaction workshop at IEEE VIS in 2019, modeled an analyst’s observations to help them navigate large text corpora, using social media data on emotional distress. Scholastic, presented at UIST 2022, is a visual analytics tool for inductive and interpretive text analysis that uses familiar graphical metaphors with interactive topic modeling and document clustering to support sampling and coding.

Building community. We co-organized the CSCW 2019 workshop on qualitative methods for CSCW, which brought researchers together around the challenges of making qualitative work legible and of scaling it responsibly. In 2020 we also recruited a postdoctoral researcher to extend the project’s qualitative and design investigations.

Why it matters

As large language models have made AI assistance in qualitative coding widespread, the questions this project raised have become pressing: where AI can help, where it threatens the interpretive judgment qualitative scholarship rests on, and how to design tools that keep analysts in charge. The project’s findings and tools remain a reference point for that conversation, and the Identity Lab’s later work on AI in sensitive contexts grew in part from it.

Researchers

Jed Brubaker, Danielle Albers Szafir, Casey Fiesler, Michael Paul, Jessica Feuston, Jialun Aaron Jiang, Kandrea Wade, Matt-Heun Hong, Lauren Marsh, Janet Ruppert, Harshini Priya Muthukrishnan

Funding

National Science Foundation, CHS: Medium: Scaling Qualitative Inductive Analysis through Computational Methods, Award 1764089.

Publications

PACM HCI
Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker
Proceedings of the ACM on Human-Computer Interaction 5, 1: Article 94
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3449168
Qualitative inductive methods are widely used in CSCW and HCI research for their ability to generatively discover deep and contextualized insights, but these inherently manual and human-resource-intensive processes are often infeasible for analyzing large corpora. Researchers have been increasingly interested in ways to apply qualitative methods to "big" data problems, hoping to achieve more generalizable results from larger amounts of data while preserving the depth and richness of qualitative methods. In this paper, we describe a study of qualitative researchers’ work practices and their challenges, with an eye towards whether this is an appropriate domain for human-AI collaboration and what successful collaborations might entail. Our findings characterize participants’ diverse methodological practices and nuanced collaboration dynamics, and identify areas where they might benefit from AI-based tools. While participants highlight the messiness and uncertainty of qualitative inductive analysis, they still want full agency over the process and believe that AI should not interfere. Our study provides a deep investigation of task delegability in human-AI collaboration in the context of qualitative analysis, and offers directions for the design of AI assistance that honor serendipity, human agency, and ambiguity.
CSCW
Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis Jessica L. Feuston and Jed R. Brubaker
Proceedings of the ACM on Human-Computer Interaction 5, CSCW2: 1–25
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3479856
Large datasets or ’big data’ corpora are typically the domain of quantitative scholars, who work with computational tools to derive numerical and descriptive insights. However, recent work asks how computational tools and other technologies, such as AI, can support qualitative scholars in developing deep and complex insights from large amounts of data. Addressing this question, Jiang et al. found that qualitative scholars are generally opposed to incorporating AI in their practices of data analysis. In this paper, we provide nuance to these earlier findings, showing that the stage of qualitative analysis matters for how scholars believe AI can and should be used. Through interviews with 15 CSCW and HCI qualitative researchers, we explore how AI can be included throughout different stages of qualitative analysis. We find that qualitative scholars are amenable to working with AI in diverse ways, such as for data exploration and coding, as long as it assists rather than automates their analytic work practice. Based on our analysis, we discuss how incorporating AI into qualitative research can shift some analytic practices, and how designing for human-AI collaboration in qualitative analysis necessitates considering tradeoffs in scale, abstraction, and task delegation. CCS Concepts: • Human-centered computing → Collaborative and social computing theory, concepts and paradigms; Collaborative and social computing design and evaluation methods.