Search results

2023

A complex logic query in a knowledge graph refers to a query expressed in logic form that conveys a complex meaning, such as where did the Canadian Turing award winner graduate from? Knowledge graph reasoning-based applications, such as dialogue systems and interactive search engines, rely on the ability to answer complex logic queries as a fundamental task. In most knowledge graphs, edges are typically

Machine learning

Personalization for BERT-based discriminative speech recognition rescoring

Jari Kolehmainen, Yi Gu, Aditya Gourav, Prashanth Gurunath Shivakumar, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

2023

Recognition of personalized content remains a challenge in end-to-end speech recognition. We explore three novel approaches that use personalized content in a neural rescoring step to improve recognition: gazetteers, prompting, and a cross-attention based encoder-decoder model. We use internal de-identified en-US data from interactions with a virtual voice assistant supplemented with personalized named

Machine learning

DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer

Goeric Huybrechts, Srikanth Ronanki, Xilai Li, Hadis Nosrati, Sravan Bodapati, Katrin Kirchhoff

2023

Conformer-based end-to-end models have become ubiquitous these days and are commonly used in both streaming and non-streaming automatic speech recognition (ASR). Techniques like dual-mode and dynamic chunk training helped unify streaming and non-streaming systems. However, there remains a performance gap between streaming with a full and limited past context. To address this issue, we propose the integration

Conversational AI / Natural-language processing

RankFormer: Listwise learning-to-rank using listwide labels

Maarten Buyl, Paul Missault, Pierre-Antoine Sondag

2023

Web applications where users are presented with a limited selection of items have long employed ranking models to put the most relevant results first. Any feedback received from users is typically assumed to reflect a relative judgement on the utility of items, e.g. a user clicking on an item only implies it is better than items not clicked in the same ranked list. Hence, the objectives optimized in Learning-to-Rank

Search and information retrieval

Sequential kernelized independence testing

Sasha Podkopaev, Patrick Bloebaum, Shiva Kasiviswanathan, Aaditya Ramdas

2023

Independence testing is a classical statistical problem that has been extensively studied in the batch setting when one fixes the sample size before collecting data. However, practitioners often prefer procedures that adapt to the complexity of a problem at hand instead of setting sample size in advance. Ideally, such procedures should (a) stop earlier on easy tasks (and later on harder tasks), hence making

Machine learning

From hypergraph energy functions to hypergraph neural networks

Yuxin Wang, Quan Gan, Xipeng Qiu, Xuanjing Huang, David Paul Wipf

2023

Hypergraphs are a powerful abstraction for representing higher-order interactions between entities of interest. To exploit these relationships in making downstream predictions, a variety of hypergraph neural network architectures have recently been proposed, in large part building upon precursors from the more traditional graph neural network (GNN) literature. Somewhat differently, in this paper we begin

Machine learning

Thompson sampling with diffusion generative prior

Yu-Guan Hsieh, Shiva Kasiviswanathan, Branislav Kveton, Patrick Bloebaum

2023

In this work, we initiate the idea of using denoising diffusion models to learn priors for online decision making problems. We specifically focus on bandit meta-learning, aiming to learn a policy that performs well across bandit tasks of a same class. To this end, we train a diffusion model that learns the underlying task distribution and combine Thompson sampling with the learned prior to deal with new

Machine learning

Multi-task off-policy learning from bandit feedback

Joey Hong, Branislav Kveton, Manzil Zaheer, Sumeet Katariya, Mohammad Ghavamzadeh

2023

Many practical problems involve solving similar tasks. In recommender systems, the tasks can be users with similar preferences; in search engines, the tasks can be items with similar affinities. To learn statistically efficiently, the tasks can be organized in a hierarchy, where the task affinity is captured using an unknown latent parameter. We study the problem of off-policy learning for similar tasks

Machine learning

eCat: An end-to-end model for multi-speaker TTS & many-to-many fine-grained prosody transfer

Ammar Abbas, Sri Karlapati, Bastian Schnell, Penny Karanasou, Marcel Granero Moya, Amith Nagaraj, Ayman Boustati, Nicole Peinelt, Alexis Moinet, Thomas Drugman

2023

We present eCat, a novel end-to-end multi-speaker model capable of: a) generating long-context speech with expressive and contextually appropriate prosody, and b) performing fine-grained prosody transfer between any pair of seen speakers. eCat is trained using a two-stage training approach. In Stage I, the model learns speaker-independent word-level prosody representations in an end-to-end fashion from

Conversational AI / Natural-language processing

Jiahang Li

Software Dev Engineer Intern

On the initialization of graph neural networks

Jiahang Li, Yakun Song, Xiang Song, David Paul Wipf

2023

Graph Neural Networks (GNNs) have displayed considerable promise in graph representation learning across various applications. The core learning process requires the initialization of model weight matrices within each GNN layer, which is typically accomplished via classic initialization methods such as Xavier initialization. However, these methods were originally motivated to stabilize the variance of hidden

Machine learning

Knowledge distillation on joint task end-to-end speech translation

Khandokar Md. Nayem, Ran Xue, Ching-Yun (Frannie) Chang, Akshaya Vishnu Kudlu Shanbhogue

2023

An End-to-End Speech Translation (E2E-ST) model takes input audio in one language and directly produces output text in another language. The model requires to learn both speech-to-text modality conversion and translation tasks, which demands a large architecture for effective learning of this joint task. Yet, to the best of our knowledge, we are the first to optimize compression of E2E-ST models. In this

Conversational AI / Natural-language processing

Resolving ambiguities in text-to-image generative models

Ninareh Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Rahul Gupta

2023

Natural language often contains ambiguities that can lead to misinterpretation and miscommunication. While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and commonsense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate the Text-to-image

Computer vision

Multi-VALUE: A framework for cross-dialectal English NLP

Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, Diyi Yang

ACL 2023 Workshop on SustaiNLP

2023

Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard

Conversational AI / Natural-language processing

Prompting language models improves performance in imbalanced setting

Jay Mohta

2023

Prompting is a widely adopted technique for fine-tuning large language models. Recent research by Scao and Rush (2021) has demonstrated its effectiveness in improving few-shot learning performance compared to vanilla fine-tuning and also showed that prompting and vanilla fine tuning achieves similar performance in high data regime (∼> 2000 samples). This paper investigates the impact of imbalanced data

Conversational AI / Natural-language processing

S2vNTM: Semi-supervised vMF neural topic modeling

Weijie Xu, Jay Desai, Srinivasan Sengamedu, "SHS", Xiaoyu Jiang, Francis Iannacci

ICLR 2023 Workshop on Practical Machine Learning for Developing Countries

2023

Language model based methods are powerful techniques for text classification. However, the models have several shortcomings. (1) It is difficult to integrate human knowledge such as keywords. (2) It needs a lot of resources to train the models. (3) It relied on large text data to pretrain. In this paper, we propose Semi-Supervised vMF Neural Topic Modeling (S2vNTM) to overcome these difficulties. S2vNTM

Conversational AI / Natural-language processing

A unified framework of graph information bottleneck for robustness and membership privacy

Enyan Dai, Limeng Cui, Zhengyang Wang, Xianfeng Tang, Yinghan Wang, Monica Cheng, Bing Yin, Suhang Wang

Priyanka Sen, Sandeep Mavadia, Amir Saffari

2023

Graph Neural Networks (GNNs) have achieved great success in modeling graph-structured data. However, recent works show that GNNs are vulnerable to adversarial attacks which can fool the GNN model to make desired predictions of the attacker. In addition, training data of GNNs can be leaked under membership inference attacks. This largely hinders the adoption of GNNs in high-stake domains such as e-commerce

Security, privacy, and abuse prevention

Knowledge graph-augmented language models for complex question answering

ACL 2023 Workshop on Natural Language Reasoning and Structured Explanations

2023

Large language models have shown impressive abilities to reason over input text, however, they are prone to hallucinations. On the other hand, end-to-end knowledge graph question answering (KGQA) models output responses grounded in facts, but they still struggle with complex reasoning, such as comparison or ordinal questions. In this paper, we propose a new method for complex question answering where we

Conversational AI / Natural-language processing

Don’t stop self-supervision: Accent adaptation of speech representations via residual adapters

Anshu Bhatia, Sanchit Sinha, Saket Dingliwal, Karthik Gopalakrishnan, Sravan Bodapati, Katrin Kirchhoff

ICLR 2023 Workshop on Trustworthy Machine Learning for Healthcare

2023

Speech representations learned in a self-supervised fashion from massive unlabeled speech corpora have been adapted successfully toward several downstream tasks. However, such representations may be skewed toward canonical data characteristics of such corpora and perform poorly on atypical, nonnative accented speaker populations. With the state-of-the-art HuBERT model as a baseline, we propose and investigate

Conversational AI / Natural-language processing

Implicit query parsing for product search

Chen Luo, Rahul Goutam, Haiyang Zhang, Chao Zhang, Yangqiu Song, Bing Yin

SIGIR 2023

2023

Query Parsing aims to extract product attributes, such as color, brand, and product type, from search queries. These attributes play a crucial role in search engines for tasks such as matching, ranking, and recommendation. There are two types of attributes: explicit attributes that are mentioned explicitly in the search query, and implicit attributes that are mentioned implicitly. Existing works on query

Search and information retrieval

KDSTM: Neural semi-supervised topic modeling with knowledge distillation

Weijie Xu, Billy Jiang, Jay Desai, Bin Han, Fuqin Yan, Francis Iannacci

ICLR 2022 Workshop on Practical Machine Learning for Developing Countries

2023

In text classification tasks, fine tuning pretrained language models like BERT and GPT-3 yields competitive accuracy; however, both methods require pretraining on large text datasets. In contrast, general topic modeling methods possess the advantage of analyzing documents to extract meaningful patterns of words without the need of pretraining. To leverage topic modeling’s unsupervised insights extraction

Conversational AI / Natural-language processing

Explaining multiclass classifiers with categorical values: A case study in radiography

Luca Franceschi, Cemre Zor, Bilal Zafar, Gianluca Detommaso, Cédric Archambeau, Tamas Madl, Michele Donini, Matthias Seeger

2023

Explainability of machine learning methods is of fundamental importance in healthcare to calibrate trust. A large branch of explainable machine learning uses tools linked to the Shapley value, which have nonetheless been found difficult to interpret and potentially misleading. Taking multiclass classification as a reference task, we argue that a critical issue in these methods is that they disregard the

Machine learning

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

Weijie Xu, Xiaoyu Jiang, Srinivasan Sengamedu, "SHS", Francis Iannacci, Jinjin Zhao

ACL Findings 2023

2023

Recently, Neural Topic Models (NTM), inspired by variational autoencoders, have attracted a lot of research interest; however, these methods have limited applications in the real world due to the challenge of incorporating human knowledge. This work presents a semi-supervised neural topic modeling method, vONTSS, which uses von Mises-Fisher (vMF) based variational autoencoders and optimal transport. When

Conversational AI / Natural-language processing

Differentially private optimization on large model at small cost

Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis

Devang Kulshreshtha, Saket Dingliwal, Brady Houston, Sravan Bodapati

2023

Differentially private (DP) optimization is the standard paradigm to learn large neural networks that are accurate and privacy-preserving. The computational cost for DP deep learning, however, is notoriously heavy due to the per-sample gradient clipping. Existing DP implementations are 2 ∼ 1000× more costly in time and space complexity than the standard (non-private) training. In this work, we develop a

Machine learning

Multilingual contextual adapters to improve custom word recognition in low-resource languages

2023

Connectionist Temporal Classification (CTC) models are popular for their balance between speed and performance for Automatic Speech Recognition (ASR). However, these CTC models still struggle in other areas, such as personalization towards custom words. A recent approach explores Contextual Adapters, wherein an attention-based biasing model for CTC is used to improve the recognition of custom entities.

Conversational AI / Natural-language processing

Scaling laws for discriminative speech recognition rescoring models

Yi Gu, Prashanth Gurunath Shivakumar, Jari Kolehmainen, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

ACL 2023 Workshop on Biomedical Natural Language Processing (BioNLP)

2023

Recent studies have found that model performance has a smooth power-law relationship, or scaling laws, with training data and model size, for a wide range of problems. These scaling laws allow one to choose nearly optimal data and model sizes. We study whether this scaling property is also applicable to second-pass rescoring, which is an important component of speech recognition systems. We focus on RescoreBERT

Machine learning

Understanding and answering incomplete questions

Angus Addlesee, Marco Damonte

CUI 2023

2023

Voice assistants interrupt people when they pause mid-question, a frustrating interaction that requires the full repetition of the entire question again. This impacts all users, but particularly people with cognitive impairments. In human-human conversation, these situations are recovered naturally as people understand the words that were uttered. In this paper we build answer pipelines which parse incomplete

Conversational AI / Natural-language processing

ADEQA: A question-answer based approach for joint ADE-suspect extraction using sequence-to-sequence transformers

Vinayak Arannil, Tomal Deb, Atanu Roy

2023

Early identification of Adverse Drug Events (ADE) is critical for taking prompt actions while introducing new drugs into the market. These ADEs information are available through various unstructured data sources like clinical study reports, patient health records, social media posts, etc. Extracting ADEs and the related suspect drugs using machine learning is a challenging task due to the complex linguistic

Conversational AI / Natural-language processing

Parameter-efficient low-resource dialogue state tracking by prompt tuning

Mingyu Derek Ma, Jiun-Yu Kao, Shuyang Gao, Arpit Gupta, Di Jin, Tagyoung Chung, Violet Peng

2023

Dialogue state tracking (DST) is an important step in dialogue management to keep track of users’ beliefs. Existing works fine-tune all language model (LM) parameters to tackle the DST task, which requires significant data and computing resources for training and hosting. The cost grows exponentially in the real-world deployment where dozens of fine-tuned LM are used for different domains and tasks. To

Conversational AI / Natural-language processing

Solving string constraints using SAT

Kevin Lotz, Amit Goel, Bruno Dutertre, Benjamin Kiesl-Reiter, Soonho Kong, Rupak Majumdar, Dirk Nowotka

CAV 2023

2023

String solvers are automated-reasoning tools that can solve combinatorial problems over formal languages. They typically operate on restricted first-order logic formulas that include operations such as string concatenation, substring relationship, and regular expression matching. String solving thus amounts to deciding the satisfiability of such formulas. While there exists a variety of different string

Automated reasoning

An AMR-based link prediction approach for document-level event argument extraction

Yuqing Yang, Qipeng Guo, Xiangkun Hu, Yue Zhang, Qipeng Guo, Zheng Zhang

Dantong Liu, Kaushik Pavani, Sunny Dasgupta

2023

Recent works have introduced Abstract Meaning Representation (AMR) for Document-level Event Argument Extraction (Doc-level EAE), since AMR provides a useful interpretation of complex semantic structures and helps to capture long-distance dependency. However, in these works AMR is used only implicitly, for instance, as additional features or training signals. Motivated by the fact that all event structures

Conversational AI / Natural-language processing

Small is the new big: Pre-finetuned compact models are better for asynchronous active learning

ACL 2023 Workshop on SustaiNLP

2023

We examine the effects of model size and pre-finetuning in an active learning setting where classifiers are trained from scratch on 14 binary and 3 multi-class text classification tasks. We make an important observation that, in realistic active learning settings, where the human annotator and the active learning system operate in asynchronous mode, a compact pre-finetuned 1-layer transformer model with

Machine learning

Sunny Dasgupta

Adaptation approaches for nearest neighbor language models

Rishabh Bhardwaj, George Polovets, Monica Sunkara

ACL Findings 2023

2023

Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, by leveraging large-scale neighborhood retrieval over external memory datastores. However, there has been little investigation into adapting such models for new domains. This work attempts to fill that gap and suggests the following approaches for adapting kNN-LMs — 1) adapting the underlying

Conversational AI / Natural-language processing

Text-to-image editing by image information removal

Zhongping Zhang, Skyler Zheng, Jacob Zhiyuan Fang, Bryan A. Plummer

CVPR 2023 Workshop on AI for Content Creation

2023

Diffusion models have demonstrated impressive performance in text-guided image generation. To leverage the knowledge of text-guided image generation models in image editing, current approaches either fine-tune the pretrained models using the input image (e.g., Imagic) or incorporate structure information as additional constraints into the pretrained models (e.g., ControlNet). However, fine-tuning large-scale

Computer vision

Understanding and constructing latent modality structures in multi-modal representation learning

Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Tran, Yi Xu, Belinda Zeng, Trishul Chilimbi

CVPR 2023

2023

Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream task performance. In this paper, based on an information-theoretic argument, we first prove that exact modality

Computer vision

Aditya Thokala

Data Scientist

David Jung

Applied Scientist

Automated analyses of IOT event monitoring systems

Andrew Apicellii, Sam Bayless, Ankush Das, Andrew Gacek, Dhiva Jaganathan, Saswat Padhi, Vaibhav Sharma, Mike Whalen, Raveesh Yadav

CAV 2023

2023

AWS IoT Events is an AWS service that makes it easy to respond to events from IoT sensors and applications. Detector models in AWS IoT Events enable customers to monitor their equipment or device fleets for failures or changes in operation and trigger actions when such events occur. If these models are incorrect, they may become out-of-sync with the actual state of the equipment causing customers to be

Automated reasoning

Raveesh Yadav

Senior Software Dev Engineer

Dhiva Jaganathan

SDE

Multilingual context-based pronunciation learning for text-to-speech

Giulia Comini, Sam Ribeiro, Fan Yang, Heereen Shim, Jaime Lorenzo Trueba

2023

Phonetic information and linguistic knowledge are an essential component of a Text-to-speech (TTS) front-end. Given a language, a lexicon can be collected offline and Grapheme-toPhoneme (G2P) relationships are usually modeled in order to predict the pronunciation for out-of-vocabulary (OOV) words. Additionally, post-lexical phonology, often defined in the form of rule-based systems, is used to correct pronunciation

Conversational AI / Natural-language processing

Learning when to trust which teacher for weakly supervised ASR

Aakriti Agrawal, Milind Rao, Anit Kumar Sahu, Gopinath (Nath) Chennupati, Andreas Stolcke

2023

Automatic speech recognition (ASR) training can utilize multiple experts as teacher models, each trained on a specific domain or accent. Teacher models may be opaque in nature since their architecture may be not be known or their training cadence is different from that of the student ASR model. Still, the student models are updated incrementally using the pseudo-labels generated independently by the expert

Conversational AI / Natural-language processing

Streaming speech-to-confusion network speech recognition

Denis Filimonov, Prabhat Pandey, Ariya Rastrow, Ankur Gandhe, Andreas Stolcke

2023

In interactive automatic speech recognition (ASR) systems, low-latency requirements limit the amount of search space that can be explored during decoding, particularly in end-to-end neural ASR. In this paper, we present a novel streaming ASR architecture that outputs a confusion network while maintaining limited latency, as needed for interactive applications. We show that 1-best results of our model are

Conversational AI / Natural-language processing

Querywise fair learning to rank through multi-objective optimization

Debabrata Mahapatra, Chaosheng Dong, Michinari Momma

2023

In Learning-to-Rank (LTR) problems, the task of delivering relevant search results and allocating fair exposure to items of a protected group can conflict. Previous works in Fair LTR have attempted to resolve this by combining the objectives of relevant ranking and fair ranking into a single linear combination, but this approach is limited by the nonconvexity of the objective functions and can result in

Information and knowledge management

Multi-label learning to rank through multi-objective optimization

Debabrata Mahapatra, Chaosheng Dong, Yetian Chen, Michinari Momma

Rohit Paturi, Sundararajan Srinivasan, Xiang Li

2023

Learning to Rank (LTR) technique is ubiquitous in Information Retrieval systems, especially in search ranking applications. The relevance labels used to train ranking models are often noisy measurements of human behavior, such as product ratings in product searches. This results in non-unique ground truth rankings and ambiguity. To address this, Multi-Label LTR (MLLTR) is used to train models using multiple

Machine learning

Lexical speaker error correction: Leveraging language models for speaker diarization error correction

2023

Speaker diarization (SD) is typically used with an automatic speech recognition (ASR) system to ascribe speaker labels to recognized words. The conventional approach reconciles outputs from independently optimized ASR and SD systems, where the SD system typically uses only acoustic information to identify the speakers in the audio stream. This approach can lead to speaker errors especially around speaker

Machine learning

Model-internal slot-triggered biasing for domain expansion in neural transducer ASR models

Edie Lu, Philip Harding, Kanthashree Mysore Sathyendra, Sibo Tong, Xuandi Fu, Jing Liu, Feng-Ju (Claire) Chang, Simon Wiesler, Grant Strimel

2023

Personal rare word recognition is an important yet challenging task for end-to-end speech recognition. Contextual biasing has demonstrated success in tackling this problem. Though effective in improving rare word recognition, these mechanisms can lead to errors due to false-biasing while facing further challenges when attempting to expand them to many domains. To address these limitations, in this work

Conversational AI / Natural-language processing

A static evaluation of code completion by large language models

Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng LI, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, Bing Xiang

WACV 2023 Workshop on Pretraining Large Vision and Multimodal Models

2023

Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the contrary, static

Conversational AI / Natural-language processing

Scalable and accurate self-supervised multimodal representation learning without aligned video and text data

Vladislav Lialin, Stephen Rawls, David Chan, Shalini Ghosh, Anna Rumshisky, Wael Hamza

2023

Scaling up weakly-supervised datasets has shown to be highly effective in the image-text domain and has contributed to most of the recent state-of-the-art computer vision and multimodal neural networks. However, existing large-scale video-text datasets and mining techniques suffer from several limitations, such as the scarcity of aligned data, the lack of diversity in the data, and the difficulty of collecting

Computer vision

Selective biasing with trie-based contextual adapters for personalised speech recognition using neural transducers

Philip Harding, Sibo Tong, Simon Wiesler

2023

Neural transducer ASR models achieve state of the art accuracy on many tasks, however rare word recognition poses a particular challenge as models often fail to recognise words that occur rarely, or not at all, in the training data. Methods of contextual biasing, where models are dynamically adapted to bias their outputs towards a given list of relevant words and phrases, have been shown to be effective

Conversational AI / Natural-language processing

Conmer: Streaming Conformer without self-attention for interactive voice assistants

Martin Radfar, Paulina Lyskawa, Brandon Trujillo, Yi Xie, Kai Zhen, Jahn Heymann, Denis Filimonov, Grant Strimel, Nathan Susanj, Athanasios Mouchtaris

2023

Conformer is an extension of transformer-based neural ASR models whose fundamental component is the self-attention module. In this paper, we show that we can remove the self-attention module from Conformer and achieve the same or even better recognition performance for utterances whose length is up to around 10 seconds. This is particularly important for streaming interactive voice assistants as input is

Conversational AI / Natural-language processing

Paulina Lyskawa

Language Engineer

Effective training of attention-based contextual biasing adapters with synthetic audio for personalised ASR

Burin Naowarat, Philip Harding, Pasquale D'Alterio, Sibo Tong, Bashar Awwad Shiekh Hasan

2023

Contextual biasing (CB) is an effective approach for contextualising hidden features of neural transducer ASR models to improve rare word recognition. CB relies on relatively large quantities of relevant human annotated natural speech during training, limiting its effectiveness in low-resource scenarios. In this work, we propose a novel approach that reduces the reliance on real speech by using synthesised

Conversational AI / Natural-language processing

Understanding disrupted sentences using underspecified abstract meaning representation

Angus Addlesee, Marco Damonte

2023

Voice assistant accessibility is generally overlooked as today’s spoken dialogue systems are trained on huge corpora to help them understand the ‘average’ user. This raises frustrating barriers for certain user groups as their speech shifts from the average. People with dementia pause more frequently mid-sentence for example, and people with hearing impairments may mispronounce words learned post-diagnosis

Conversational AI / Natural-language processing

IGB: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research

Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, Wen-mei Hwu

2023

Graph neural networks (GNNs) have shown high potential for a variety of real-world, challenging applications, but one of the major obstacles in GNN research is the lack of large-scale flexible datasets. Most existing public datasets for GNNs are relatively small, which limits the ability of GNNs to generalize to unseen data. The few existing large-scale graph datasets provide very limited labeled data.

Machine learning

On-device constrained self-supervised speech representation learning for keyword spotting via knowledge distillation

Gene-Ping Yang, Yue Gu, Qingming Tang, Dongsu Du, Yuzong Liu

ACL 2023 Workshop on SustaiNLP

2023

Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we proposed a knowledge distillation-based self-supervised speech representation learning (S3RL) architecture for on-device keyword spotting. Our approach used a teacher-student framework to

Conversational AI / Natural-language processing

KGQA without retraining

Nick McKenna, Priyanka Sen

2023

Popular models for Knowledge Graph Question Answering (KGQA), including semantic parsing and End-to-End (E2E) models, decode into a constrained space of KG relations. Al-though E2E models accommodate novel entities at test-time, this constraint means they cannot access novel relations, requiring expensive and time-consuming retraining whenever a new relation is added to the KG. We propose KG-Flex, a new

Conversational AI / Natural-language processing

Knowledge-augmented language model prompting for zero-shot knowledge graph question answering

Jinheon Baek, Alham Fikri Aji, Amir Saffari

ACL 2023 Workshop on Matching Entities

2023

Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowl-edge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we pro-pose

Conversational AI / Natural-language processing

Towards building a robust toxicity predictor

Dmitriy Bespalov, Sourav Bhabesh, Yi Xiang, Yanjun (Jane) Qi

Akshaya Vishnu Kudlu Shanbhogue, Ran Xue, Soumya Saha, Daniel Zhang, Ashwin Ganesan

2023

Recent NLP literature pays little attention to the robustness of toxicity language predictors, while these systems are most likely to be used in adversarial contexts. This paper presents a novel adversarial attack, ToxicTrap, introducing small word-level perturbations to fool SOTA text classifiers to predict toxic text samples as benign. ToxicTrap exploits greedy based search strategies to enable fast and

Conversational AI / Natural-language processing

Recipes for sequential pre-training of multilingual encoder and seq2seq models

Saleh Soltan, Andy Rosenbaum, Tobias Falke, Qin Lu, Anna Rumshisky, Wael Hamza

ACL Findings 2023, ACL 2023 Workshop on SustaiNLP

2023

Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages; however, training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency by initializing one model from the other. (1) Extracting the encoder from a seq2seq model, we show it underperforms a Masked Language Modeling (MLM) encoder, particularly on sequence labeling

Conversational AI / Natural-language processing

Qin Lu

Sonja Remien

Principal Research Scientist

Krishnaditya Rathore

Data Scientist

Siddhant Chauhan

Data Scientist

Ray Hensberger

Senior Manager, Data Science

Sam Gupta

Applied Scientist

Chung Chi Leung

Data Scientist

Daniel Chiu

Applied Scientist

Junshen Xu

Applied Scientist

Improving low resource speech translation with data augmentation and ensemble strategies

IWSLT 2023

2023

This paper describes the speech translation system submitted as part of the IWSLT 2023 shared task on low resource speech translation. The low resource task aids in building models for language pairs where the training corpus is limited. In this paper, we focus on two language pairs, namely, Tamasheq-French (Tmh→Fra) and Marathi-Hindi (Mr→Hi) and implement a speech translation system that is unconstrained

Conversational AI / Natural-language processing

Record deduplication for entity distribution modeling in ASR transcripts

Tianyu Huang, Chung Hoon Hong, Carl Wivagg, Kanna Shimizu

2023

Voice digital assistants must keep up with trending search queries. We rely on a speech recognition model using contextual biasing with a rapidly updated set of entities, instead of frequent model retraining, to keep up with trends. There are several challenges with this approach: (1) the entity set must be frequently reconstructed, (2) the entity set is of limited size due to latency and accuracy trade-offs

Conversational AI / Natural-language processing

Developing and validating automated scoring for an audio constructed response simulation

MQ Liu, Denver McNeney, John Capman, Shane Lowery, Matthew Kitching, Anjali Nimbkar, Tony Boyce

Personnel Psychology

2023

We evaluated the effectiveness of machine learning (ML) and natural language processing (NLP) for automatically scoring a simulation requiring audio-based constructed responses. We administered the simulation to 3,174 recent new professional-level hires working in a large multinational technology company. Human subject matter experts (SMEs) scored each response using behaviorally anchored rating scales

Conversational AI / Natural-language processing

Anjali Nimbkar

Sr. Technical Program Manager

SWING: Balancing coverage and faithfulness for dialogue summarization

Kung-Hsiang Huang, Siffi Singh, Xiaofei Ma, Wei Xiao, Feng Nan, Nick Dingwall, William Wang, Kathleen McKeown

EACL 2023

2023

Missing information is a common issue of dialogue summarization where some information in the reference summaries is not covered in the generated summaries. To address this issue, we propose to utilize natural language inference (NLI) models to improve coverage while avoiding introducing factual inconsistencies. Specifically, we use NLI to compute fine-grained training signals to encourage the model to

Conversational AI / Natural-language processing

Exploiting abstract meaning representation for open-domain question answering

Cunxiang Wang, Zhikun Xu, Qipeng Guo, Xiangkun Hu, Xuefeng Bai, Zheng Zhang, Yue Zhang

ACL Findings 2023

2023

The Open-Domain Question Answering (ODQA) task involves retrieving and subsequently generating answers from fine-grained relevant passages within a database. Current systems leverage Pre-trained Language Models (PLMs) to model the relationship between questions and passages. However, the diversity in surface form expressions can hinder the model’s ability to capture accurate correlations, especially within

Conversational AI / Natural-language processing

Causal triplet: An open challenge for intervention-centric causal representation learning

Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, Francesco Locatello

CLeaR 2023

2023

Recent years have seen a surge of interest in learning high-level causal representations from low-level image pairs under interventions. Yet, existing efforts are largely limited to simple synthetic settings that are far away from real-world problems. In this paper, we present Causal Triplet, a causal representation learning benchmark featuring not only visually more complex scenes, but also two crucial

Machine learning

Causal discovery with score matching on additive models with arbitrary noise

Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, Francesco Locatello

CLeaR 2023

2023

Causal discovery methods are intrinsically constrained by the set of assumptions needed to ensure structure identifiability. Moreover additional restrictions are often imposed in order to simplify the inference task: this is the case for the Gaussian noise assumption on additive nonlinear models, which is common to many causal discovery approaches. In this paper we show the shortcomings of inference under

Machine learning

Dynamic inference with grounding based vision and language models

Burak Uzkent, Amanmeet Garg, Wentao Zhu, Keval Doshi, Jingru Yi, Andy Wang, Mohamed Omar

CVPR 2023

2023

Transformers have been recently utilized for vision and language tasks successfully. For example, recent image and language models with more than 200M parameters have been proposed to learn visual grounding in the pre-training step and show impressive results on downstream vision and language tasks. On the other hand, there exists a large amount of computational redundancy in these large models which skips

Computer vision

KILM: Knowledge injection into encoder-decoder language models

Yan XU, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, Dilek Hakkani-Tür

2023

Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters. To enhance this implicit knowledge, we propose Knowledge Injection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through continued pre-training. This is done without architectural modifications

Conversational AI / Natural-language processing

Can you teach a computer to smell? Osmo is trying

Sean O'Neill

June 12, 2023

On the left is a color map (the CIE 1931 color space chromaticity diagram), similar colors lie near each other. On the right is Osmo’s Principal Odor Map, individual molecules (grey points) are found nearer to each other if they are predicted to smell similar.

The company’s work, supported by the Amazon Alexa Fund, has relevant applications for areas from perfumes to disease detection.

Machine learning

Unsupervised melody-to-lyric generation

Yufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Chenyang Tao, Gunnar Sigurdsson, Wenbo Zhao, Tagyoung Chung, Jing Huang, Violet Peng

2023

Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical interest and more challenging than unconstrained lyric generation as the music imposes additional constraints onto the lyrics. The training data is limited as most songs are copyrighted, resulting in models that underfit the complicated cross-modal relationship between

Conversational AI / Natural-language processing

Petter Nilsson

Applied Scientist

Kexin Liang

Data Scientist

ICASSP 2023

ICASSP 2023 will offer a comprehensive technical program presenting all the latest developments in research and technology for signal processing and its applications. Featuring world-class oral and poster sessions, keynotes, plenaries and perspective talks, tutorials, special sessions, exhibitions and demonstrations, industry workshops and entrepreneurial presentations, it is expected to attract leading researchers and global industry pioneers.

Conversational AI / Natural-language processing

June 4 - 10, 2023

Rhodes Island, Greece

Website

Gayathri Sridhar

Data Scientist

DFEE: Interactive DataFlow execution and evaluation kit

Han He, Song Feng, Daniele Bonadiman, Yi Zhang, Saab Mansour

2023

We present DFEE, an interactive DataFlow Execution and Evaluation toolkit that supports execution, visualization and benchmarking of semantic parsers given dialogue input and backend database. We demonstrate the system via a complex dialog task: event scheduling that involves temporal reasoning. It also supports diagnosing the parsing results via a friendly interface that allows developers to examine dynamic

Conversational AI / Natural-language processing

Multilingual robust contrastive pretraining

Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Saab Mansour

2023

To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP). RCP

Conversational AI / Natural-language processing

Synthetic data for model selection

Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Matan Fintz, Gérard Medioni

ICML 2023, ICLR 2021 Workshop on Synthetic Data Generation (Quality, Privacy, Bias)

2023

Recent breakthroughs in synthetic data generation approaches made it possible to produce highly photorealistic images which are hardly distinguishable from real ones. Furthermore, synthetic generation pipelines have the potential to generate an unlimited number of images. The combination of high photorealism and scale turn synthetic data into a promising candidate for improving various machine learning

Computer vision

Measuring and mitigating local instability in deep neural networks

Arghya Datta, Subhrangshu Nandi, Jingcheng Xu, Greg Ver Steeg, He Xie, Anoop Kumar, Aram Galstyan

ACL Findings 2023

2023

Deep Neural Networks (DNNs) are becoming integral components of real world services relied upon by millions of users. Unfortunately, architects of these systems can find it difficult to ensure reliable performance as irrelevant details like random initialization can unexpectedly change the outputs of a trained system with potentially disastrous consequences. We formulate the model stability problem by studying

Conversational AI / Natural-language processing

RobustQA: Benchmarking the robustness of domain adaptation for open-domain question answering

Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Wang, Zhiheng Huang, Bing Xiang, Dan Roth

ACL Findings 2023

2023

Open-domain question answering (ODQA) is a crucial task in natural language processing. A typical ODQA system relies on a retriever module to select relevant contexts from a large corpus for a downstream reading comprehension model. Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models’ generalizability across diverse domains, as models are trained and evaluated

Conversational AI / Natural-language processing

Hybrid hierarchical retrieval for open-domain question answering

Manoj Ghuhan Arivazhagan, Lan Liu, Peng Qi, Xinchi Chen, William Wang, Zhiheng Huang

ACL Findings 2023

2023

Retrieval accuracy is crucial to the performance of open-domain question answering (ODQA) systems. Recent work has demonstrated that dense hierarchical retrieval (DHR), which retrieves document candidates first and then relevant passages from the refined document set, can significantly outperform the single stage dense passage retriever (DPR). While effective, this approach requires document structure information

Conversational AI / Natural-language processing

SCOTT: Self-consistent chain-of-thought distillation

Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren

2023

Large language models (LMs) beyond a certain scale, demonstrate the emergent capability of generating free-text rationales for their predictions via chain-of-thought (CoT) prompting. While CoT can yield dramatically improved performance, such gains are only observed for sufficiently large LMs. Even more concerning, there is little guarantee that the generated rationales are consistent with LM’s predictions

Conversational AI / Natural-language processing

Sampling bias in NLU models: Impact and mitigation

Zefei Li, Anil Ramakrishna, Anna Rumshisky, Andy Rosenbaum, Saleh Soltan, Rahul Gupta

2023

Natural Language Understanding (NLU) systems such as chatbots or virtual assistants have seen a significant rise in popularity in recent times, thanks to availability of large volumes of user data. However, typical user data collected for training such models may suffer from sampling biases due to a variety of factors. In this paper, we study the impact of bias in the training data for intent classification

Conversational AI / Natural-language processing

More-inclusive speech recognition with cross-utterance rescoring

Venkatesh Ravichandran

June 09, 2023

In a top-3% paper at ICASSP, Amazon researchers adapt graph-based label propagation to improve speech recognition on underrepresented pronunciations.

Conversational AI / Natural-language processing

Translation-enhanced multilingual text-to-image generation

Yaoyiran Li, Ching-Yun (Frannie) Chang, Stephen Rawls, Ivan Vulić, Anna Korhonen

2023

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image-caption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We provide two key

Conversational AI / Natural-language processing

Sharing encoder representations across languages, domains and tasks in large-scale spoken language understanding

Jonathan Hueser, Judith Gaspers, Thomas Gueudre, Chandana Satya Prakash, Jin Cao, Daniil Sorokin, Quynh Do, Nicolas Anastassacos, Tobias Falke, Turan Gojayev, Mariusz Momotko, Denis Romasanta Rodriguez, Austin Doolittle, Kartik Balasubramaniam, Wael Hamza, Fabian Triefenbach, Patrick Lehnen

2023

Leveraging representations from pre-trained transformer-based encoders achieves state-ofthe-art performance on numerous NLP tasks. Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines). We evaluate using a larger 170M parameter BERT encoder that shares representations

Conversational AI / Natural-language processing

Kartik Balasubramaniam

Sr. SDE, Alexa AI-Natural Understanding

Denis Romasanta Rodriguez

Sr. Software Dev Engineer, Alexa AI-Natural Understanding

Austin Doolittle

Senior Software Dev Engineer, Alexa AI-Natural Understanding

Mariusz Momotko

Principal SDE, Alexa AI-Natural Understanding

MLSys 2023

The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.

Machine learning

June 4 - 8, 2023

Miami, Florida

Website

AVEN-GR: Attribute value extraction and normalization using product graphs

Donato Crisostomi, Thomas Ricatte

2023

Getting a good understanding of the user intent is vital for e-commerce applications to surface the right product to a given customer query. Query Understanding (QU) systems are essential for this purpose, and many e-commerce providers are working on complex solutions that need to be data efficient and able to capture early emerging market trends. Query Attribute Understanding (QAU) is a sub-component of

Machine learning

CUPID: Curriculum learning based real-time prediction using distillation

Arindam Bhattacharya, Ankith M S, Ankit Gandhi, Vijay Huddar, Atul Saroop, Rahul Bhagat

2023

Relevance in E-commerce Product Search is crucial for providing customers with accurate results that match their query intent. With recent advancements in NLP and Deep Learning, Transformers have become the default choice for relevance classification tasks. In such a setting, the relevance model uses query text and product title as input features, and estimates if the product is relevant for the customer

Search and information retrieval

Improving grapheme-to-phoneme conversion by learning pronunciations from speech recordings

Sam Ribeiro, Giulia Comini, Jaime Lorenzo Trueba

2023

The Grapheme-to-Phoneme (G2P) task aims to convert orthographic input into a discrete phonetic representation. G2P conversion is beneficial to various speech processing applications, such as text-to-speech and speech recognition. However, these tend to rely on manually-annotated pronunciation dictionaries, which are often time-consuming and costly to acquire. In this paper, we propose a method to improve

Conversational AI / Natural-language processing

Distillation strategies for discriminative speech recognition rescoring

Prashanth Gurunath Shivakumar, Jari Kolehmainen, Yi Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

2023

Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by exploiting the knowledge from masked language model pretraining. Further, fine-tuning with discriminative loss such as minimum word error rate (MWER) has shown to perform better than likelihood-based loss. Streaming applications

Machine learning

Toward more accurate and generalizable evaluation metrics for task-oriented dialogs

Abi Komma, Nagesh Panyam, Timothy Leffel, Anuj Goyal, Angeliki Metallinou, Spyros Matsoukas, Aram Galstyan

2023

Measurement of interaction quality is a critical task for the improvement of spoken dialog systems. Existing approaches to dialog quality estimation either focus on evaluating the quality of individual turns, or collect dialog-level quality measurements from end users immediately following an interaction. In contrast to these approaches, we introduce a new dialoglevel annotation workflow called Dialog Quality

Conversational AI / Natural-language processing

Benchmarking diverse-modal entity linking with generative models

Sijia Wang, Alexander Li, Henry Zhu, Sheng Zhang, Chung-Wei Hang, Pramuditha Perera, JIE MA, William Wang, Zhiguo Wang, Vittorio Castelli, Bing Xiang, Patrick Ng

ACL Findings 2023

2023

Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding, or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed

Conversational AI / Natural-language processing

Theoretical guarantees of learning ensembling strategies with applications to time series forecasting

Hilaf Hasson, Danielle Maddix Robinson, Bernie Wang, Gaurav Gupta, Youngsuk Park

2023

Ensembling is among the most popular tools in machine learning (ML) due to its effectiveness in minimizing variance and thus improving generalization. Most ensembling methods for black-box base learners fall under the umbrella of “stacked generalization,” namely training an ML algorithm that takes the inferences from the base learners as input. While stacking has been widely applied in practice, its theoretical

Machine learning

Context-aware query rewriting for improving users’ search experience on e-commerce websites

Simiao Zuo, Qingyu Yin, Haoming Jiang, Shaohui Xi, Bing Yin, Chao Zhang, Tuo Zhao

2023

E-commerce queries are often short and ambiguous. Consequently, query understanding often uses query rewriting to disambiguate user input queries. While using e-commerce search tools, users tend to enter multiple searches, which we call context, before purchasing. These history searches contain contextual insights about users’ true shopping intents. Therefore, modeling such contextual information is critical

Search and information retrieval

Shaohui Xi

Software Development Engineer, Search Experience Science

Controlled text generation with hidden representation transformations

Vaibhav Kumar, Hana Koorehdavoudi, Masud Moshtaghi, Amita Misra, Ankit Chadha, Emilio Ferrara

ACL Findings 2023

2023

We propose CHRT (Control Hidden Representation Transformation) — a controlled language generation framework that steers large language models to generate text pertaining to certain attributes (such as toxicity). CHRT gains attribute control by modifying the hidden representation of the base model through learned transformations. We employ a contrastive-learning framework to learn these transformations that

Conversational AI / Natural-language processing

CONTRACLM: Contrastive learning for causal language model

Nihal Jain, Dejiao Zhang, Wasi Ahmad, Zijian Wang, Feng Nan, Xiaopeng LI, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, Bing Xiang

2023

Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations

Conversational AI / Natural-language processing

Few shot rationale generation using self-training with dual teachers

Aditya Srikanth Veerubhotla, Lahari Poddar, Jun Yin, Gyuri Szarvas, Sharanya Eswaran

ACL Findings 2023

2023

Self-rationalizing models that also generate a free-text explanation for their predicted labels are an important tool to build trustworthy AI applications. Since generating explanations for annotated labels is a laborious and costly process, recent models rely on large pretrained language models (PLMs) as their backbone and few-shot learning. In this work we explore a self-training approach leveraging both

Conversational AI / Natural-language processing

Yong Yu

Amazon Research Award recipient

Noah Smith

Amazon Research Award recipient

Spyros Makridakis

Amazon Research Award recipient

Efficient Shapley values estimation by amortization for text classification

Alan Yang, Fan Yin, He He, Kai-Wei Chang, Xiaofei Ma, Bing Xiang

2023

Despite the popularity of Shapley Values in explaining neural text classification models, computing them is prohibitive for large pretrained models due to a large number of model evaluations. In practice, Shapley Values are often estimated with a small number of stochastic model evaluations. However, we show that the estimated Shapley Values are sensitive to random seed choices — the top-ranked features

Conversational AI / Natural-language processing

Characterizing and measuring linguistic dataset drift

Tyler A. Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth

2023

NLP models often degrade in performance when real world data distributions differ markedly from training data. However, existing dataset drift metrics in NLP have generally not considered specific dimensions of linguistic drift that affect model performance, and they have not been validated in their ability to predict model performance at the individual example level, where such metrics are often used in

Conversational AI / Natural-language processing

Scalable and safe remediation of defective actions in self-learning conversational systems

Sarthak Ahuja, Mohammad Kachuee, Fateme Sheikholeslami, Weiqing Liu, Jae Do

2023

Off-Policy reinforcement learning has been a driving force for the state-of-the-art conversational AIs leading to more natural humanagent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system

Conversational AI / Natural-language processing

Weiqing Liu

Software Development Engineer, Intelligent Decisions

Code-switched text synthesis in unseen language pairs

I-Hung Hsu, Avik Ray, Shubham Garg, Nanyun Peng, Jing Huang

ACL Findings 2023

2023

Existing efforts on text synthesis for codeswitching mostly require training on codeswitched texts in the target language pairs, limiting the deployment of the models to cases lacking code-switched data. In this work, we study the problem of synthesizing codeswitched texts for language pairs absent from the training data. We introduce GLOSS, a model built on top of a pre-trained multilingual machine translation

Conversational AI / Natural-language processing

Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale

Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, Dan Roth

2023

Language models have been shown to perform better with an increase in scale on a wide variety of tasks via the in-context learning paradigm. In this paper, we investigate the hypothesis that the ability of a large language model to in-context learn-perform a task is not uniformly spread across all of its underlying components. Using a 66 billion parameter language model (OPT-66B) across a diverse set of

Conversational AI / Natural-language processing

Diable: Efficient dialogue state tracking as operations on tables

Pietro Lesci, Yoshinari Fujinuma, Momchil Hardalov, Chao Shang, Lluis Marquez

ACL Findings 2023

2023

Sequence-to-sequence state-of-the-art systems for dialogue state tracking (DST) use the full dialogue history as input, represent the current state as a list with all the slots, and generate the entire state from scratch at each dialogue turn. This approach is inefficient, especially when the number of slots is large and the conversation is long. In this paper, we propose Diable, a new task formalisation

Conversational AI / Natural-language processing

Human transcription quality improvement

Jian Gao, Hanbo Sun, Cheng Cao, Zheng Du

2023

High quality transcription data is crucial for training automatic speech recognition (ASR) systems. However, the existing industry-level data collection pipelines are expensive to researchers, while the quality of crowdsourced transcription is low. In this paper, we propose a reliable method to collect speech transcriptions. We introduce two mechanisms to improve transcription quality: confidence estimation

Conversational AI / Natural-language processing

Niccolo Guglielmo

Data Scientist

Jessie Jebaraj

Data Scientist

Elad Mezuman

Applied Scientist

RAMP: Retrieval and attribute-marking enhanced prompting for attribute-controlled translation

Gabriele Sarti, Phu Mon Htut, Xing Niu, Benjamin Hsu, Anna Currey, Georgiana Dinu, Maria Nădejde

2023

Attribute-controlled translation (ACT) is a subtask of machine translation that involves controlling stylistic or linguistic attributes (like formality and gender) of translation outputs. While ACT has garnered attention in recent years due to its usefulness in real-world applications, progress in the task is currently limited by dataset availability, since most prior approaches rely on supervised methods

Conversational AI / Natural-language processing

XSEMPLR: Cross-lingual semantic parsing in multiple natural languages and meaning representations

Yusen Zhang, Jun Wang, Zhiguo Wang, Rui Zhang

2023

Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs) such as SQL, lambda calculus, and logic forms. However, existing CLSP models are separately proposed and evaluated on datasets of limited tasks and applications, impeding a comprehensive and unified evaluation of CLSP on a diverse range of NLs and MRs. To this end, we present

Conversational AI / Natural-language processing

ECG-QALM: Entity-controlled synthetic text generation using contextual Q&A for NER

Karan Aggarwal, Henry Jin, Aitzaz Ahmad

ACL Findings 2023

2023

Named Entity Recognition (NER) state-of-the-art methods require high-quality labeled datasets. Issues such as scarcity of labeled data, under-representation of entities, and privacy concerns with using sensitive data for training can be significant barriers. Generating synthetic data to train models is a promising solution to mitigate these problems. We propose ECG-QALM, a contextual question and answering

Conversational AI / Natural-language processing

LI-RAGE: Late interaction retrieval augmented generation with explicit signals for open-domain table question answering

Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adrià de Gispert, Gonzalo Iglesias

2023

Recent open-domain TableQA models are typically implemented as retriever-reader pipelines. The retriever component is usually a variant of the Dense Passage Retriever, which computes the similarities between questions and tables based on a single representation of each. These fixed vectors can be insufficient to capture fine-grained features of potentially very big tables with heterogeneous row/column information

Search and information retrieval

Unified contextual query rewriting

Yingxue Zhou, Jie Hao, Mukund Rungta, Yang Liu, Eunah Cho, Xing Fan, Yanbin Lu, Vishal Vasudevan, Kellen Gillespie, Zeynab Raeesy, Sawyer Shen, Edward Guo, Gokhan Tur

2023

Query rewriting (QR) is an important technique for user friction reduction (i.e. recovering ASR error or system error) and contextual carryover (i.e. ellipsis and co-reference) in conversational AI systems. Recently, generation-based QR models have achieved promising results on these two tasks separately. Although these two tasks have many similarities such as they both use the previous dialogue along with

Conversational AI / Natural-language processing

An inner table retriever for robust table question answering

Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adrià de Gispert, Gonzalo Iglesias

2023

Recent years have witnessed the thriving of pretrained Transformer-based language models for understanding semi-structured tables, with several applications, such as Table Question Answering (TableQA). These models are typically trained on joint tables and surrounding natural language text, by linearizing table content into sequences comprising special tokens and cell information. This yields very long

Conversational AI / Natural-language processing

Generate then select: Open-ended visual question answering guided by world knowledge

Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henry Zhu, Yuhao Zhang, Alexander Hanbo Li, William Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang

ACL Findings 2023

2023

The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge. Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. However, these methods suffer from low knowledge coverage caused by PLM bias – the tendency to generate certain tokens

Computer vision

Entity contrastive learning in a large-scale virtual assistant system

Jonathan Rubin, Jason Crowley, George Leung, Morteza Ziyadi, Maria Minakova