Search results

We present a very simple method for extending pretrained machine translation metrics to incorporate document-level context. We apply our method to four popular metrics: BERTScore, Prism, COMET, and the reference-free metric COMET-QE. We evaluate our document-level metrics on the MQM annotations from the WMT 2021 metrics shared task and find that the document-level metrics outperform their sentence-level

In this work, we propose an effective and intuitive termination criterion for BO that automatically stops the procedure if it is sufficiently close to the global optimum. Our key insight is that the discrepancy between the true objective (predictive performance on test data) and the computable target (validation performance) suggests stopping once the suboptimality in optimizing the target is dominated

Semantic parsing is an important NLP problem, particularly for voice assistants such as Alexa and Google Assistant. State-of-the-art (SOTA) semantic parsers are seq2seq architectures based on large language models that have been pretrained on vast amounts of text. To better leverage that pretraining, recent work has explored a reformulation of semantic parsing whereby the output sequences are themselves

The Alexa Voice Service (AVS) enables developers to integrate Alexa directly into their products, bringing the convenience of voice control to any connected device. AVS provides developers with access to a suite of resources to build Alexa-enabled products, including APIs, hardware development kits, software development kits, and documentation.

Leveraging the characteristics of convolutional layers, neural networks are extremely effective for pattern recognition tasks. However in some cases, their decisions are based on unintended information leading to high performance on standard benchmarks but also to a lack of generalization to challenging testing conditions and unintuitive failures. Recent work has termed this ”shortcut learning” and addressed

Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and

The FoodOrdering dataset is a task-oriented parsing dataset in the food-ordering domain with utterances and annotations derived from the menus of five venues characteristic of that business vertical: burgers, burritos, coffees, pizzas, and subs.

A 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B), which achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French

Automatic speech recognition (ASR) is increasingly being used in specialized domains such as medical ASR and news transcription. Owing to the lack of high quality annotated speech data in such domains, off-the-shelf models are commonly employed by fine-tuning on domain-specific data. This poses a significant challenge in transcribing long-tail expressions and out-of-vocabulary (OOV) named entities. On the

Current question answering (QA) systems primarily consider the single-answer scenario, where each question is assumed to be paired with one correct answer. However, in many real-world QA applications, multiple answer scenarios arise where consolidating answers into a comprehensive and non-redundant set of answers is a more efficient user interface. In this paper, we formulate the problem of answer consolidation

Project Description This is a test framework for the bias bounties project. Getting Started as a Bounty Hunter If you are interacting with this codebase as a "bounty hunter", you'll need to have a way to run Jupyter notebooks. The easiest way to do this is to download Anaconda, which will also manage all of your python packages for you. See here for installation instructions: https://docs.anaconda.com/anaconda

This module implements a set of functions to perform MAXENT from a causal perspective. The code here can be used to reproduce the results in the publication Obtaining Causal Information by Merging Datasets with MAXENT. The parts of the plots using KCI are, unfortunately, not available. To reproduce the results in the article create a python 3.6+ environment, pip install all the requirements.txt file and

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased (Freitag et al., 2021; Isabelle et al., 2017). In particular, gender accuracy in translation (Choubey et al., 2021; Saunders and Byrne, 2020) can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MTGenEval

Robots operating in human spaces must be able to engage in natural language interaction, both understanding and executing instructions, and using conversation to resolve ambiguity and correct mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human–human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates

Skill Components are “ready to use” experiences that you can easily add to your skills, and configure them according to your needs. Each component is a collection of skill primitives, such as VUI dialogs & intents, Alexa Presentation Language (APL) documents, skill code, skill connection tasks, and skill events. They bring together design best practices and pre-built voice experiences, which solve for a

We present GRILLBot, a multi-modal task-oriented voice assistant to guide users through complex real-world tasks for the Alexa TaskBot Challenge. An effective TaskBot has to guide a user through a long and complex task, be engaging, and help solve problems along the way. GRILLBot achieves this in the domains of cooking and home improvement by helping search over a large task corpus with mixed- initiative

Tartan is a multi-domain task-oriented bot that assists users with two different tasks: 1. cooking with recipes from Whole Foods and 2. doing projects from WikiHow. The bot’s system is divided into two stages. In the first stage, the bot assists the user in identifying a task that they want to do. In the second stage, the bot guides the users through sequences of step-by-step instructions. We developed

Since its inception in 2016, the Alexa Prize program has enabled hundreds of university students to explore and compete to develop conversational agents through the SocialBot Grand Challenge. The goal of the challenge is to build agents capable of conversing coherently and engagingly with humans on popular topics for 20 minutes, while achieving an average rating of at least 4.0/5.0. However, as conversational

We describe QuakerBot, a dialog system that helps users with household tasks and a participant in the Alexa Prize TaskBot Challenge. QuakerBot can process a variety of user requests, search for instructions from web resources such as wikiHow or Whole Foods Market recipes, answer related questions, and so on. Its components simultaneously consist of large language models with an impressive few-shot performance

As people learn to interact with AI assistants such as Alexa, their needs change and become more complex. Alexa must evolve accordingly, and offer more sophisticated experiences, which require addressing increasingly complex AI research challenges. In the last few years, Amazon has offered university teams a chance to partner with Amazon scientists in order push the boundaries of the state of the art in