Marginalization & Safety

Connected digital technologies, like social media platforms and mobile applications, offer many benefits to users of marginalized identities. They provide avenues for identity exploration, access to otherwise inaccessible resources, and opportunities for building community. However, they can also put users at risk for negative, harmful, and damaging encounters. Harassment by other users, incidental interactions with violent content, and even outing by the system itself all present potential dangers to using these types of technologies–dangers that can even harmfully impact people’s offline lives. These unsafe situations are often more precarious for users of (multiply) marginalized identities, such as LGBTQ users, people of color, and women. We are working towards developing better understandings of safety and harm in digital spaces, as well as researching these experiences to better understand how to mitigate their affects.

Publications

CSCW
A Framework of Severity for Harmful Content Online Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R. Brubaker
Proceedings of the ACM on Human-Computer Interaction 5, CSCW2: Article 368
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3479512
The proliferation of harmful content on online social media platforms has necessitated empirical understandings of experiences of harm online and the development of practices for harm mitigation. Both understandings of harm and approaches to mitigating that harm, often through content moderation, have implicitly embedded frameworks of prioritization-what forms of harm should be researched, how policy on harmful content should be implemented, and how harmful content should be moderated. To aid efforts of better understanding the variety of online harms, how they relate to one another, and how to prioritize harms relevant to research, policy, and practice, we present a theoretical framework of severity for harmful online content. By employing a grounded theory approach, we developed a framework of severity based on interviews and card-sorting activities conducted with 52 participants over the course of ten months. Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We describe how our framework can be applied to both research and policy settings towards deeper understandings of specific forms of harm (e.g., harassment) and prioritization frameworks when implementing policies encompassing many forms of harm.
Journal
Understanding international perceptions of the severity of harmful content online Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker
PLOS ONE 16, 8: 1–22
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1371/journal.pone.0256762
Online social media platforms constantly struggle with harmful content such as misinformation and violence, but how to effectively moderate and prioritize such content for billions of global users with different backgrounds and values presents a challenge. Through an international survey with 1,696 internet users across 8 different countries across the world, this empirical study examines how international users perceive harmful content online and the similarities and differences in their perceptions. We found that across countries, the perceived severity consistently followed an exponential growth as the harmful content became more severe, but what harmful content were perceived as more or less severe varied significantly. Our results challenge platform content moderation’s status quo of using a one-size-fits-all approach to govern international users, and provide guidance on how platforms may wish to prioritize and customize their moderation of harmful content.
CSCW
Best Paper Diversity and Inclusion Recognition
How We’ve Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R. Brubaker
Proceedings of the ACM on Human-Computer Interaction 4, CSCW1: Article 58
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3392866
Race and gender have long sociopolitical histories of classification in technical infrastructures-from the passport to social media. Facial analysis technologies are particularly pertinent to understanding how identity is operationalized in new technical systems. What facial analysis technologies can do is determined by the data available to train and evaluate them with. In this study, we specifically focus on this data by examining how race and gender are defined and annotated in image databases used for facial analysis. We found that the majority of image databases rarely contain underlying source material for how those identities are defined. Further, when they are annotated with race and gender information, database authors rarely describe the process of annotation. Instead, classifications of race and gender are portrayed as insignificant, indisputable, and apolitical. We discuss the limitations of these approaches given the sociohistorical nature of race and gender. We posit that the lack of critical engagement with this nature renders databases opaque and less trustworthy. We conclude by encouraging database authors to address both the histories of classification inherently embedded into race and gender, as well as their positionality in embedding such classifications.
CSCW
Best Paper
How Computers See Gender: An Evaluation of Gender Classification in Commercial Facial Analysis and Image Labeling Services Morgan Klaus Scheuerman, Jacob M Paul, and Jed R. Brubaker
Proceedings of the ACM on Human-Computer Interaction 3, CSCW: Article 144
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3359246
Investigations of facial analysis (FA) technologies-such as facial detection and facial recognition-have been central to discussions about Artificial Intelligence’s (AI) impact on human beings. Research on automatic gender recognition, the classification of gender by FA technologies, has raised potential concerns around issues of racial and gender bias. In this study, we augment past work with empirical data by conducting a systematic analysis of how gender classification and gender labeling in computer vision services operate when faced with gender diversity. We sought to understand how gender is concretely conceptualized and encoded into commercial facial analysis and image labeling technologies available today. We then conducted a two-phase study: (1) a system analysis of ten commercial FA and image labeling services and (2) an evaluation of five services using a custom dataset of diverse genders using self-labeled Instagram images. Our analysis highlights how gender is codified into both classifiers and data standards. We found that FA services performed consistently worse on transgender individuals and were universally unable to classify non-binary genders. In contrast, image labeling often presented multiple gendered concepts. We also found that user perceptions about gender performance and identity contradict the way gender performance is encoded into the computer vision infrastructure. We discuss our findings from three perspectives of gender identity (self-identity, gender performativity, and demographic identity) and how these perspectives interact across three layers: the classification infrastructure, the third-party applications that make use of that infrastructure, and the individuals who interact with that software. We employ Bowker and Star’s concepts of "torque" and "residuality" to further discuss the social implications of gender classification. We conclude by outlining opportunities for creating more inclusive classification infrastructures and datasets, as well as with implications for policy.
CSCW
"Coming Out Okay": Community Narratives for LGBTQ Identity Recovery Work Brianna Dym, Jed R. Brubaker, Casey Fiesler, and Bryan Semaan
Proceedings of the ACM on Human-Computer Interaction 3, CSCW: Article 154
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3359256
Online communities provide support for those who are vulnerable, such as LGBTQ people while coming out. Research shows that social support and personal narrative construction are important when recovering from personal crises and traumatic events. As an online community focused on writing fanfiction and also consisting of a large number of LGBTQ members, transformative fandom provides an opportunity to examine the relationship between support, crisis, and narrative. Through an interview study with 31 LGBTQ fanfiction authors, our findings mirror Herman’s model of trauma recovery: these spaces self-organize to support recovery work through constructing "community narratives" that help LGBTQ people establish safety when exploring their identity and build LGBTQ support structures without publicly outing themselves before they are ready, challenge stereotypes, and support others through reshaping existing media that perpetuate inaccurate or harmful LGBTQ narratives. These online communities embody "selective visibility"–that is, though not specifically designed as support structures for identity work and recovery, their design allows people to selectively find and create communities of support for stigmatized issues that they might be unable to safely seek out in other spaces. Based on lessons learned, we generate insights that can inform the design of safe support spaces online.
CSCW
Diversity and Inclusion Recognition
Safe Spaces and Safe Places: Unpacking Technology-Mediated Experiences of Safety and Harm with Transgender People Morgan Klaus Scheuerman, Stacy M. Branham, and Foad Hamidi
Proceedings of the ACM on Human-Computer Interaction 2, CSCW Article 155
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3274424
Transgender individuals in the United States face significant threats to interpersonal safety; however, there has as yet been relatively little research in the HCI and CSCW communities to document transgender individuals’ experiences of technology-mediated safety and harm. In this study, we interviewed 12 transgender and non-binary individuals to understand how they find, create, and navigate safe spaces using technology. Managing safety was a universal concern for our transgender participants, and they experienced complex manifestations of harm through technology. We found that harmful experiences for trans users could arise as targeted or incidental affronts, as sourced from outsiders or insiders, and as directed against individuals or entire communities.. Notably, some violations implicated technology design, while others tapped broader social dynamics. Reading our findings through the notions of ’space" and ’place," we unpack challenges and opportunities for building safer futures with transfolk, other vulnerable users, and their allies.
ASSETS
"Is Someone There? Do They Have a Gun": How Visual Information About Others Can Improve Personal Safety Management for Blind Individuals Stacy M. Branham, Ali Abdolrahmani, William Easley, Morgan Scheuerman, Erick Ronquillo, and Amy Hurst
International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2017)
- Abstract
- PDF
- Reference
- BibTeX
- DOI: 10.1145/3132525.3132534
For decades, researchers have investigated and developed technologies that support independent navigation for people who are blind. This has led to systems that primarily aid in detecting routes, landmarks, and building features. However, there has been relatively little inquiry regarding how technologies might support navigation around and in the presence of other people. What visual information, if any, do blind navigators wish they had about people on their path? To address this question, we surveyed 58 blind and low vision individuals and interviewed 10 blind individuals. We discovered our participants were interested in using visual information about others to increase their physical safety. For example, they wanted to know if a passerby was holding a weapon, if a presumed official had a proper uniform or badge, and how to describe visual aspects of a criminal to law enforcement. This paper presents one of the only reports documenting accessibility challenges related to physical safety posed by others, including how future assistive tools can empower individuals with disabilities to more actively increase their sense of safety. We call this emerging area Personal Safety Management and contribute a set of four broad subareas that deserve further exploration by researchers and designers working within the blind and broader disabilities communities.