Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies
Authors: Sunipa Dev (she/her), Masoud Monajatipoor* (he/him), Anaelia Ovalle* (they/he/she), Arjun Subramonian* (they/them), Jeff M Phillips (he/him), Kai-Wei Chang (he/him)
This blog post contains examples of stereotypes and associations, misgendering, erasure, and other harms that could be offensive and triggering to trans and non-binary individuals.
Recently, the advance of AI has enabled many applications based on natural language processing technology. For example, large language models, trained on vast amount of texts collected from books, news, and the internet, have significantly boosted the performance of language processing systems. However, several studies (Bender et al. 2021, Blodgett et al. 2020) have pointed out significant limitations of these models.
Language models can perpetrate harms such as the cyclical erasure of non-binary gender identities. Some recent works attempt to mitigate these harms by building task-specific datasets that are not restricted to binary gender and building metrics that on extension, could potentially measure biases against all genders. While such works that intentionally inject real-world or artificially-created data of non-binary people into binary-gendered datasets are well-intentioned, they could benefit from a broader perspective of harms as perceived by non-binary persons to avoid mischaracterizing non-binary genders as a single gender or perpetuating biases through non-intersectional training examples, i.e. examples that do not capture the interconnected nature of social identities.
In our EMNLP 2021 paper Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies, we explain the complexity of gender and language around it, and survey non-binary persons to understand representational and allocational harms associated with the treatment of gender as binary in English language technologies. We also detail how current language representations (e.g., GloVe, BERT) capture and perpetuate these harms and related challenges that need to be acknowledged and addressed for representations to equitably encode gender information.
Survey on Harms
We conducted an anonymous survey to understand representational and allocational harms associated with the skewed treatment of gender in three English NLP applications: named entity recognition, coreference resolution, and machine translation. We focused our survey on non-binary persons who have familiarity with AI.
Respondents identified representational harms in these applications such as systematically mistagging neopronouns and singular they as non-person entities, not recognizing neopronouns, and translating a word from one language into an offensive term in another language.
Respondents also identified allocational harms such as resume scanning systems throwing out resumes from non-binary persons for not having a recognizable name, undercounting instances of discrimination against non-binary persons, and incorrectly processing gender in medical documents, leading to incorrect care.
Data and Modeling Challenges
We explored how current language representations (e.g. GloVe, BERT) capture and perpetuate representational harms. To replicate our experiments, you can visit https://github.com/uclanlp/harms-challenges-non-binary-representation-NLP.
Dataset Skews
The large text dumps often used to build language representations have severe skews with respect to gender and gender-related concepts. Just observing pronoun usage, English Wikipedia text (March 2021 dump), which comprises 4.5 billion tokens, has over 15 million mentions of he and 4.8 million of she, but only 4.9 million of they (including plural they) and 4.5 thousand of xe (including the company XE). Furthermore, the usage of non-binary-associated pronouns (i.e. neopronouns and gendered pronouns that are not he or she) were mostly not meaningful with respect to gender.
Some corpora contain instances of meaningfully used non-binary-associated pronouns, such as the Non-Binary Wiki (https://nonbinary.wiki/wiki/Main_Page). However, via manual evaluation, we found that they have two drawbacks: (i) the narratives are mostly short biographies and lack the diversity of sentence structures as seen in the rest of Wikipedia, and (ii) they have the propensity to be dominated by Western cultures, resulting in the further sparsification of diverse narratives of non-binary persons.
Representational Erasure and Biased Associations in GloVe
After training GloVe embeddings on recent Wikipedia articles, we identified that the singular pronouns he and she, as well as their possessive forms, have semantically meaningful neighbors. However, due to underrepresentation, the same is not true for neopronouns xe and ze; we found a lack of meaningful encodings of non-binary-associated words, with the closest neighbors being acronyms and Polish words, respectively. Furthermore, since neopronouns are not well-embedded, we found disparate sentiment associations between binary versus non-binary-associated pronouns, gendered words, and proxies (e.g. male, female versus transman, genderqueer, etc.)
Representational Erasure and Biased Associations in BERT
We evaluated BERT’s ability to distinguish between singular pronouns (e.g. he, singular they) and plural pronouns (e.g. plural they). Using samples from both Wikipedia and another source of non-binary biographical content, we found that, to some extent, BERT is able to distinguish between they(p) and they(s) with 67.7% accuracy, but not as well as for he and they(p) (83.3% accuracy). This indicates that for BERT, they(s) is not as distinguishable from they(p) as a binary-associated pronoun like he.
Additionally, using sentence templates like “[Alex] [went to] the [hospital] for [PP][appointment]. [MASK] was [feeling sick].”, we evaluated BERT’s ability to predict the masked pronoun in the second sentence in each template for various possessive pronouns (PP) in the first sentence. BERT scores high on his and her, but scores lower on their; on xyr and hir, BERT’s accuracy drops even more. This demonstrates that BERT can explicitly misgender non-binary persons even when context is provided for correct pronoun usage.
Related Work and Future Directions
A bulk of social bias studies on language technologies have focused on binary gender and the stereotypes associated with masculine and feminine attributes.
![An example of binary gender bias when applying coreference resolution to “The physician hired the secretary because [he/she] was overwhelmed with clients.” The coreference resolution system predicts that “he” corefers with “physician” and that “she” corefers with “secretary.”](https://miro.medium.com/v2/format:webp/1*nKC6gqgkQjiSK6TaAPSbRQ.png)
While discussing binary gender bias and improving model performance are important, it is important to reshape our understanding of gender in language technologies in a more accurate, inclusive, non-binary manner.
While there are efforts to better include non-binary genders in language technologies, it is questionable if these technologies should define gender in discrete quantities, when in reality, gender is of a fluid and flexible nature. Models currently do not account for the mutability of gender and the language around it. Bucketing gender in immutable, discrete units and trying to represent each, would inevitably result in marginalization of sections of the population to varied extents. As our survey catalogs how pronounced the harms of being consistently misgendered and diminished are, we encourage future work to carefully examine how (and if) to define and model gender in language representations and applications.
This work sets the interdisciplinary stage for rethinking and addressing challenges with inclusively modeling gender in language technologies. Any viable solution cannot simply be a quick fix or patch, but must rely on a bottom-up approach involving affected persons system-wide, such as in annotation and human-in-the-loop mechanisms. Simultaneously, research into monitoring language technologies over time to detect harms against non-binary individuals is critical. It is further paramount to transparently communicate the performance of language technologies for non-binary persons and possible harms. In the case of harm, non-binary individuals must be able to obtain valid recourse to receive a more favorable outcome, as well as have the opportunity to provide feedback on the model’s output and have a human intervene.
