Entity-level factual consistency of abstractive text summarization

By Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
2021
Download Copy BibTeX
Copy BibTeX
A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.
Research areas

Latest news

US, CA, Pasadena
The Amazon Center for Quantum Computing in Pasadena, CA, is looking to hire a Fabrication R&D Scientist with experience in semiconductor process development who will aid in Amazon’s effort to bring cloud quantum computing services to its worldwide customer base. You will join a multi-disciplinary team of scientists, and hardware and software engineers working at the forefront of quantum computing. Through your work inside and outside of the cleanroom environment in the fabrication research and development group, you will solve problems related to developing next-generation quantum processors. Candidates must have a demonstrated background in sound scientific and engineering principles, and must have excellent data analysis, bias for action, problem solving, and communication skills, and be highly motivated and curious to research and learn new technical topics as needed. As a Fab R&D scientist you will be expected to work on new ideas and stay abreast of novel approaches in fabricating and packaging superconducting quantum processors. Working effectively within a team environment is critical. Diverse Experiences Amazon values diverse experiences. Even if you do not meet all the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Export Control Requirement Due to applicable export control laws and regulations, candidates must be either a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum, or be able to obtain a US export license. If you are unsure if you meet these requirements, please apply and Amazon will review your application for eligibility. Key job responsibilities Responsibilities include developing and optimizing processes to fabricate high-coherence superconducting qubits; developing advanced 3DI interconnect and routing technologies for integrating superconducting quantum technologies; analyzing inline metrology and electrical test data; developing and maintaining integration documentation, design rules, and standard operating procedures; interacting with project leads to provide feedback that continuously improves different processes; staying updated with the latest advancements and industry trends in process integration and apply knowledge to improve processes and drive innovation providing technical guidance and support to junior colleagues, fostering a collaborative and knowledge-sharing work environment. A day in the life The candidate will develop novel technologies using micro-/nano-fabrication techniques inside the cleanroom (independently or in collaboration with other scientists, engineers, and technicians) for next-generation quantum computing. Outside the cleanroom, the candidate will plan experiments, analyze data, and conceive future innovations.
US, CA, San Francisco
Amazon Industrial Robotics is on a mission to redefine the future of automation — and we're looking for exceptional talent to help lead the way. We are building the next generation of advanced robotic systems that seamlessly blend cutting-edge AI, sophisticated control systems, and novel mechanical design to create adaptable, intelligent automation solutions capable of operating safely alongside humans in dynamic, real-world environments. At Amazon Industrial Robotics, we leverage the power of machine learning, artificial intelligence, and advanced robotics to solve some of the most complex operational challenges at a scale unlike anywhere else in the world. Our fleet of robots spans hundreds of facilities globally, working in sophisticated coordination to deliver on our promise of customer excellence — and we're just getting started. As a Sr. Applied Scientist in Robot Perception, you will be at the forefront of this transformation. You will develop and deploy state-of-the-art perception algorithms that enable robots to truly understand and interact with the physical world — bridging the gap between theoretical research and realworld impact. Bringing deep expertise in Computer Vision and a nuanced understanding of the capabilities and limitations of modern Vision-Language Models (VLMs), you will innovate boldly and push the boundaries of what's possible. Our vision for the Perception layer is ambitious: to enable seamless, intelligent interaction between the user, the robot, and its environment. This is a rare opportunity to work at the intersection of deep learning, large language models, and robotics — contributing to research that doesn't just advance the field, but reshapes it. You will collaborate with world-class teams pioneering breakthroughs in dexterous manipulation, locomotion, and humanrobot interaction, all at an unprecedented scale. Key job responsibilities Design, develop, and deploy perception algorithms for robotics systems, including object detection, segmentation, tracking, depth estimation, and scene understanding • Lead research initiatives in computer vision, sensor fusion and 3D perception • Collaborate with cross-functional teams including robotics engineers, software engineers, and product managers to define and deliver perception capabilities • Drive end-to-end ownership of ML models — from data collection and labeling strategy to training, evaluation, and deployment • Mentor junior scientists and engineers; contribute to a culture of technical excellence • Define and track key metrics to measure perception system performance in real-world environments • Publish research findings in top-tier venues (CVPR, ICCV, ECCV, ICRA, NeurIPS, etc.) and contribute to patents A day in the life Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment • Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations • Actively contribute to brainstorming sessions on adjacent topics, bringing fresh perspectives that help peers grow and succeed — and in doing so, build lasting trust across the team • Mentor team members while maintaining significant hands-on contribution to technical solutions About the team Our Industrial Robotics Group is a diverse group of scientists and engineers passionate about building intelligent machines. We value curiosity, rigor, and a bias for action. We believe in learning from failure and iterating quickly toward solutions that matter.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
GB, London
We are looking for an Economist to work on exciting and challenging business problems related to Amazon Retail’s worldwide product assortment. You will build innovative solutions based on econometrics, machine learning, and experimentation. You will be part of a interdisciplinary team of economists, product managers, engineers, and scientists, and your work will influence finance and business decisions affecting Amazon’s vast product assortment globally. If you have an entrepreneurial spirit, you know how to deliver results fast, and you have a deeply quantitative, highly innovative approach to solving problems, and long for the opportunity to build pioneering solutions to challenging problems, we want to talk to you. Key job responsibilities * Work on a challenging problem that has the potential to significantly impact Amazon’s business position * Develop econometric models and experiments to measure the customer and financial impact of Amazon’s product assortment * Collaborate with other scientists at Amazon to deliver measurable progress and change * Influence business leaders based on empirical findings
US, WA, Bellevue
The Supply Chain Optimization Technologies (SCOT) team builds technology to automate and optimize Amazon’s supply chain of physical goods. We seek a Data Scientist with strong analytical and communication skills to join our team. SCOT manages Amazon's inventory under uncertainty of demand, pricing, promotions, supply, vendor lead times, and product life cycle. We optimize complex trade-offs between customer experience, inventory costs, fulfillment costs, fulfillment center capacity, etc. We develop sophisticated algorithms that involve learning from large amounts of data such as prices, promotions, similar products, and other data from our product catalog in order to automatically act on millions of dollars’ worth of inventory weekly and establish plans for tens of thousands of employees. As a Data Scientist, you will contribute to the research community, by working with other scientists across Amazon and our Supply Chain, as well as collaborating with academic researchers and publishing papers both internally and externally. Key job responsibilities Major responsibilities include: - Analysis of large amounts of data from different parts of the supply chain and their associated business functions - Improving upon existing machine learning methodologies by developing new data sources, developing and testing model enhancements, running computational experiments, and fine-tuning model parameters for new models - Formalizing assumptions about how models are expected to behave, creating definitions of outliers, developing methods to systematically identify these outliers, and explaining why they are reasonable or identifying fixes for them - Communicating verbally and in writing to business customers with various levels of technical knowledge, educating them about our research, as well as sharing insights and recommendations - Utilizing code (Python, R, Scala, etc.) for analyzing data and building statistical and machine learning models and algorithms A day in the life As a Data Scientist in SCOT, you will be tasked to understand and work with innovative research tools to enable the implementation of sophisticated models on big data. As a successful data scientist in the SCOT team, you are an analytical problem solver who enjoys diving into data from various businesses, is excited about investigations and algorithms, can multi-task, and can credibly interface between scientists, engineers and business stakeholders. Your expertise in synthesizing and communicating insights and recommendations to audiences of varying levels of technical sophistication will enable you to answer specific business questions and innovate for the future. Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: - Medical, Dental, and Vision Coverage - Maternity and Parental Leave Options - Paid Time Off (PTO) - 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply!
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the next-level. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Key job responsibilities * Develop, deploy, and operate scalable bioinformatics analysis workflows on AWS * Evaluate and incorporate novel bioinformatic approaches to solve critical business problems * Originate and lead the development of new data collection workflows with cross-functional partners * Partner with laboratory science teams on design and analysis of experiments About the team Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
US, CA, San Jose
Are you excited about using econometrics to make multi-million dollar decisions more Science and Data Driven? Are you interested in supporting Consumer Hardware device concepts from innovative idea inception to launch? Do you want to work on a Economics and Data Science team focused on tackling some of the hardest business questions within the Devices business at Amazon and then scaling those Statistics and Econometrics solutions via internal to Amazon tools? Then this could be the role for you! The Decision Science team owns demand estimates and pricing recommendations of concept devices before customers know they exist. We support analyses on hardware and services ranging from Echo Frames to Kindle Paperwhite to Blink Video Camera subscriptions to the Amazon Smart Plug - all prior to launch. In this role, you will develop science for high visible senior leadership decisions on new devices and services and work with a cross-functional team to apply and scale innovative science broadly. Key job responsibilities - Design, estimate, and scale Berry-Levinsohn-Pakes (BLP) random coefficients demand models to quantify consumer heterogeneity, own- and cross-price elasticities, and substitution patterns across large product markets. - Implement and optimize numerical routines—including GMM estimation, contraction mappings, and simulation-based inversion—to solve structural demand systems at scale in Python. - Develop and validate instrumental variables strategies to address price endogeneity in differentiated product markets, ensuring unbiased and robust demand parameter estimates. - Build production-grade pipelines that ingest large-scale observational datasets, estimate consumer preferences, and generate product-level demand forecasts on recurring schedules. - Collaborate with cross-functional teams including product management, marketing, and operations to translate structural model outputs—such as willingness-to-pay and competitive diversion ratios—into actionable pricing and portfolio strategies. - Advance the team's structural modeling capabilities by researching and deploying extensions to classical BLP frameworks (e.g., supply-side estimation, dynamic demand, micro-moments) and documenting approaches in clear technical reports.
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Our products are used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses a combination of econometrics, machine learning, and data science to build disruptive products for all our Advertising products. We also generate insights to guide Amazon Advertising strategy, providing direct support to senior leadership. We are looking for an experienced Economist with a deep passion for building econometric solutions and the ability to communicate data insights and scientific vision to execute on strategic projects. Key job responsibilities - Leverage econometrics and ML models to optimize advertising strategies on behalf of our customers. - Influence key business and product decisions based on insights from models you develop. - Perform hands-on analysis and modeling with enormous data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. - Work closely with software engineers on detailed requirements to productionize the models you build. - Run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Work with other scientists, software developers, and product partners to implement your solutions.
US, WA, Seattle
RISC's vision is to make Amazon Earth’s most trusted shopping destination for safe and compliant products. We do this by protecting customers from products that are unsafe, illegal, illegally marketed, controversial or otherwise in violation of Amazon's policies while enabling our Selling Partners (SPs) to offer their broadest selection of safe and compliant products. We are seeking an exceptional Applied Scientist to join a team of experts in the field of agentic AI, GenAI, Machine Learning, Software Engineers, and work together to tackle challenging problems across diverse compliance domains. We leverage and train state-of-the-art large-language-models (LLMs), multi-modal model, mixed with elegant harness engineering and SKILL building to 1) detect illegal and unsafe products across the Amazon catalog; 2) automation safety and compliance content authoring; 3) reasoning over enforcement action to provide actionable insights to Amazon sellers. We work on machine learning problems for content generation, multi-modal classification, global product taxonomy, intent detection, information retrieval, anomaly and fraud detection, agentic AI, generative AI and multi-agent system. This is an exciting and challenging position to deliver scientific innovations into production systems at Amazon-scale to make immediate, meaningful customer impacts while also pursuing ambitious, long-term research. You will work in a highly collaborative environment where you can analyze and process large amounts of image, text, unstructured and tabular data. You will work on challenging science problems that have not been solved before, conduct rapid prototyping to validate your hypothesis, and deploy your algorithmic ideas at scale. There will be something new to learn every day as we work in an environment with rapidly evolving regulations and adversarial actors looking to outwit your best ideas. Key job responsibilities • Design and evaluate state-of-the-art algorithms and approaches in content generation, multi-modal classification, global product taxonomy, intent detection, information retrieval, anomaly and fraud detection, agentic AI, generative AI and multi-agent system. • Translate product and CX requirements into measurable science problems and metrics. • Collaborate with product and tech partners and customers to validate hypothesis, drive adoption, and increase business impact • Key author in writing high quality scientific papers in internal and external peer-reviewed conferences. A day in the life • Understanding customer problems, project timelines, and team/project mechanisms • Proposing science formulations and brainstorming ideas with team to solve business problems • Writing code, and running experiments with re-usable science libraries • Reviewing labels and audit results with investigators and operations associates • Sharing science results with science, product and tech partners and customers • Writing science papers for submission to peer-review venues, and reviewing science papers from other scientists in the team. • Contributing to team retrospectives for continuous improvements • Driving science research collaborations and attending study groups with scientists across Amazon
US, NY, New York
About Sponsored Products and Brands: The Sponsored Products and Brands (SPB) organization at Amazon Ads is re-imagining the advertising landscape through industry leading generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. About Our Team: The Brand Beacon team is responsible for inventing impressions offerings for brands to increase share of voice via premium experiences, helping brands get discovered, acquire new customers and sustainably grow customer lifetime value. We build end-to-end solutions that enable brands to drive discovery, visibility and share of voice. This includes building advertiser controls, shopper experiences, monetization strategies and optimization features. We succeed when (1) shoppers discover, engage and build affinity with brands and (2) brands can grow their business at scale with our advertising products. About This Role: As a Senior Scientist for the team, you will have the opportunity to apply your deep subject matter expertise in the area of ML, LLM and GenAI models. You will invent new product experiences that enable novel advertiser and shopper experiences. This role will liaise with internal Amazon partners and work on bringing state-of-the-art GenAI models to production, and stay abreast of the latest developments in the space of GenAI and identify opportunities to improve the efficiency and productivity of the team. Additionally, you will define a long-term science vision for our advertising business, driven by our customer’s needs, and translate it into actionable plans for our team of applied scientists and engineers. This role will play a critical role in elevating the team’s scientific and technical rigor, identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. You will communicate learnings to leadership and mentor and grow Applied AI talent across org. * Develop AI solutions for advertiser and shopper experiences. Build monetization and optimization systems that leverage generative models to value and improve campaign performance. * Define a long-term science vision and roadmap for our advertising business, driven from our customers' needs, translating that direction into specific plans for applied scientists and engineering teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. * Design and conduct A/B experiments to evaluate proposed solutions based on in-depth data analyses. * Effectively communicate technical and non-technical ideas with teammates and stakeholders. * Stay up-to-date with advancements and the latest modeling techniques in the field. * Think big about the arc of development of Gen AI over a multi-year horizon and identify new opportunities to apply these technologies to solve real-world problems. #GenAI