Publications

2023

1. MEGA: Multilingual Evaluation of Generative AI

   Kabir Ahuja, Harshita Diddee, Rishav Hada, and 9 more authors

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Dec 2023

   Abs PDF Code

   Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

2. “Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text

   Rishav Hada*, Agrima Seth*, Harshita Diddee, and 1 more author

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Dec 2023

   Abs PDF

   Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline discourses. With LLMs increasingly gaining human-like fluency in text generation, gaining a nuanced understanding of the biases these systems can generate is imperative. Prior work often treats gender bias as a binary classification task. However, acknowledging that bias must be perceived at a relative scale; we investigate the generation and consequent receptivity of manual annotators to bias of varying degrees. Specifically, we create the first dataset of GPT-generated English text with normative ratings of gender bias. Ratings were obtained using Best–Worst Scaling – an efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed ranking and show that identity-attack is most closely related to gender bias. Finally, we show the performance of existing automated models trained on related concepts on our dataset.

3. Are large language model-based evaluators the solution to scaling up multilingual evaluation?

   Rishav Hada, Varun Gumma, Adrian Wynter, and 5 more authors

   Preprint Dec 2023

   PDF

2022

1. CodeFed: Federated Speech Recognition for Low-Resource Code-Switching Detection

   Chetan Madan, Harshita Diddee, Deepika Kumar, and 1 more author

   ACM Transactions Asian Low-Resource Language Information Processing (TALLIP) Aug 2022

   Abs

   One common constraint in the practical application of speech recognition is Code Switching. The issue of code-switched languages is especially aggravated in the context of Indian languages - since most massively multilingual models are trained on corpora that are not representative of the diverse set of Indian languages. An associated constraint with such systems is the privacy-intrusive nature of the applications that aim to collate such representative data. To collectively mitigate both problems, this works presents CodeFed: A federated learning-based code-switching detection model that can be deployed to collaboratively trained by leveraging private data from multiple users, without compromising their privacy. Using a representative low-resource Indic dataset, we demonstrate the superior performance of a collaboratively trained global model that is trained using federated learning on three low-resource Indic languages - Gujarati, Tamil and Telugu and draw a comparison of the model with respect to most current work in the field. Finally, to evaluate the practical realizability of the proposed system, CodeFed also discusses the system overview of the label generation architecture which may accompany CodeFed’s possible real-time deployment.

2. Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models

   Harshita Diddee, Sandipan Dandapat, Monojit Choudhury, and 2 more authors

   Conference On Machine Translation 2022 Oct 2022

   Abs PDF Code

   Leveraging shared learning through Massively Multilingual Models, state-of-the-art machine translation models are often able to adapt to the paucity of data for low-resource languages. However, this performance comes at the cost of significantly bloated models which are not practically deployable. Knowledge Distillation is one popular technique to develop competitive, lightweight models: In this work, we first evaluate its use to compress MT models focusing on languages with extremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance of the distilled models due to their dependence on priors including the amount of synthetic data used for distillation, the student architecture, training hyperparameters and confidence of the teacher models, makes distillation a brittle compression mechanism. To mitigate this, we explore the use of post-training quantization for the compression of these models. Here, we find that while distillation provides gains across some low-resource languages, quantization provides more consistent performance trends for the entire range of languages, especially the lowest-resource languages in our target set.

3. Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

   Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, and 15 more authors

   Transactions of the Association for Computational Linguistics Feb 2022

   Abs PDF Code

   We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 million sentence pairs from existing, publicly available parallel corpora, and additionally mine 37.4 million sentence pairs from the Web, resulting in a 4× increase. We mine the parallel sentences from the Web by combining many corpora, tools, and methods: (a) Web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Further, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and models are available publicly at Samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages.

4. The Six Conundrums of Building and Deploying Language Technologies for Social Good

   Harshita Diddee, Kalika Bali, Monojit Choudhury, and 1 more author

   In ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) Jun 2022

   Abs PDF

   Deployment of speech and language technology for social good (LT4SG), especially those targeted at the welfare of marginalized communities and speakers of low-resource and under-served languages, has been a prominent theme of research within NLP, Speech and the AI communities. Many researchers, especially those working in core NLP/Speech domains, rely on a combination of individual expertise, experiences or ad hoc surveys for prioritizing between language technologies that provide social good to the end-users. This has been criticized by several scholars who argue that it is critical to include the target community during the LT’s design and development process. However, prioritization of communities, languages, technologies and design approaches presents a very large set of complex challenges to the technologists, for which there are no simple or off-the-shelf solutions. In this position paper, we distill our experiential insights into six fundamental conundrums that technologists face and must resolve while deciding which LT technology to build for which community, and by using what approach. We discuss that at the root of these conundrums lie certain fundamental ethical problems of a digital-divide that can be overcome only by resolving deeper ethical dilemmas of distributive justice. We urge the community to reflect on these conundrums and leverage shared experiential insights to reconcile the intent of broadly, any Technology for Social Good, with the ground realities of its deployment.

2021

1. Towards Quantifying the Carbon Emissions of Differentially Private Machine Learning

   Rakshit Naidu, Harshita Diddee, Ajinkya Mulay, and 3 more authors

   Socially Responsible Machine Learning Workshop at ICML 2021 Jun 2021

   Abs PDF Code

   In recent years, machine learning techniques utilizing large scale datasets have achieved remarkable performance. Differential privacy, by means of adding noise, provides strong privacy guarantees for such learning algorithms. The cost of differential privacy is often a reduced model accuracy and a lowered convergence speed. This paper investigates the impact of differential privacy on learning algorithms in terms of their carbon footprint due to either longer run-times or failed experiments. Through extensive experiments, further guidance is provided on choosing the noise levels which can strike a balance between desired privacy levels and reduced carbon emissions

2020

1. PsuedoProp at SemEval-2020 Task 11: Propaganda Span Detection Using BERT-CRF and Ensemble Sentence Level Classifier

   Aniruddha Chauhan, and Harshita Diddee

   In Proceedings of the Fourteenth Workshop on Semantic Evaluation Dec 2020

   Abs PDF Code

   This paper explains our teams’ submission to the Shared Task of Fine-Grained Propaganda Detection in which we propose a sequential BERT-CRF based Span Identification model where the fine-grained detection is carried out only on the articles that are flagged as containing propaganda by an ensemble SLC model. We propose this setup bearing in mind the practicality of this approach in identifying propaganda spans in the exponentially increasing content base where the fine-tuned analysis of the entire data repository may not be the optimal choice due to its massive computational resource requirements. We present our analysis on different voting ensembles for the SLC model. Our system ranks 14th on the test set and 22nd on the development set and with an F1 score of 0.41 and 0.39 respectively.

2. CrossPriv: User Privacy Preservation Model for Cross-Silo Federated Software

   Harshita Diddee, and Bhrigu Kansra

   In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering Dec 2020

   Abs PDF Code

   The design and implementation of artificial intelligence driven software that keeps user data private is a complex yet necessary requirement in the current times. Developers must consider several ethical and legal challenges while developing services which relay massive amount of private information over a network grid which is susceptible to attack from malicious agents. In most cases, organizations adopt a traditional model training approach where publicly available data, or data specifically collated for the task is used to train the model. Specifically in the healthcare section, the operation of deep learning algorithms on limited local data may introduce a significant bias to the system and the accuracy of the model may not be representative due to lack of richly covariate training data. In this paper, we propose CrossPriv, a user privacy preservation model for cross-silo Federated Learning systems to dictate some preliminary norms of SaaS based collaborative software. We discuss the client and server side characteristics of the software deployed on each side. Further, We demonstrate the efficacy of the proposed model by training a convolution neural network on distributed data of two different silos to detect pneumonia using X-Rays whilst not sharing any raw data between the silos.