Code and datasets

To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP). RCP

Proper estimation of predictive uncertainty is fundamental in applications that involve critical decisions. Uncertainty can be used to assess the reliability of model predictions, trigger human intervention, or decide whether a model can be safely deployed in the wild. We introduce Fortuna, an open-source library for uncertainty quantification. Fortuna provides calibration methods, such as conformal prediction

Xtr-WikiQA is an Answer Sentence Selection (AS2) dataset in 9 non-English languages, proposed in our paper accepted at ACL 2023 (Findings): Cross-Lingual Knowledge Distillation for answer sentence selection in low-resource languages. This dataset is based on an English AS2 dataset, WikiQA (Original, Hugging Face). For translations, we used Amazon Translate.

TyDi-AS2 and Xtr-TyDi-AS2 are multilingual Answer Sentence Selection (AS2) datasets comprising 8 diverse languages, proposed in our paper accepted at ACL 2023 (Findings): Cross-Lingual Knowledge Distillation for answer sentence selection in low-resource languages. Both the datasets were created from TyDi-QA, a multilingual question-answering dataset. TyDi-AS2 was created by converting the QA instances in

Missing information is a common issue of dialogue summarization where some information in the reference summaries is not covered in the generated summaries. To address this issue, we propose to utilize natural language inference (NLI) models to improve coverage while avoiding introducing factual inconsistencies. Specifically, we use NLI to compute fine-grained training signals to encourage the model to

RefChecker provides a standardized assessment framework to identify subtle hallucinations present in the outputs of large language models (LLMs). Highlighted Features Finer granularity — RefChecker breaks down the claims in the LLM’s response into knowledge triplets as opposed to paragraphs, sentences, or sub-sentences. Detecting at the knowledge-triplet level will test the truthfulness of facts. Importantly

Recent work in scientific machine learning (SciML) has focused on incorporating partial differential equation (PDE) information into the learning process. Much of this work has focused on relatively “easy” PDE operators (e.g., elliptic and parabolic), with less emphasis on relatively “hard” PDE operators (e.g., hyperbolic). Within numerical PDEs, the latter problem class requires control of a type of volume

Regulations introduced General Data Protection Regulation (GDPR) in the EU or California Consumer Privacy Act (CCPA) in the US have included provisions on the right to be forgotten that mandates industry applications to remove data related to an individual from their systems. In several real world industry applications that use Machine Learning to build models on user data, such mandates require significant

Minimax-fair machine learning minimizes the error for the worst-off group. However, empirical evidence suggests that when sophisticated models are trained with standard empirical risk minimization (ERM), they often have the same performance on the worst-off group as a minimax-trained model. Our work makes this counterintuitive observation concrete. We prove that if the hypothesis class is sufficiently expressive

This is the repo for ReCode (arXiv), providing a comprehensive evaluation for the practical robustness of code generation models like CodeGEN, Incoder, GPT-J. In specific, this benchmark provides over 30 different general perturbations on docstrings, function names, and codes. The perturbations are carefully selected and implemented such that the perturbed datasets are naturally and semantically close to

KILM (short for Knowledgeable Injection into Language Model) is an approach for injecting knowledge into pre-trained language models. This repository provides the necessary code for KILM. KILM propose a new (second-step) pre-training method to inject information about entities (such as entity descriptions) into pre-trained language models. This repository contains code for Creating the dataset needed for

This is a package to apply clustering algorithms to utterances, embedded with a fine-tuned version out of the Supervised Intent Clustering package. Modern virtual assistants are trained to classify customer requests into a taxonomy of predesigned intents. Requests that fall outside of this taxonomy, however, are often unhandled and need to be clustered to define new experiences. Recently, state-of-the-art