Social
Conversational AI

A quick guide to Amazon’s 30+ papers at NAACL 2024

Although work involving large language models predominates, classical and more-general techniques remain well represented.

By Staff writer
June 7, 2024
5 min read
Share

In recent years, the fields of natural-language processing and computational linguistics, which were revolutionized a decade ago by deep learning, were revolutionized again by large language models (LLMs). Unsurprisingly, work involving LLMs, either as a subject of inquiry themselves or as tools for other natural-language-processing applications, predominates at this year’s meeting of the North American Chapter of the Association for Computational Linguistics (NAACL). This paper guide sorts Amazon’s NAACL papers into those that deal explicitly with LLMs and those that don’t — although in many cases, the ones that don’t present general techniques or datasets that could be used with either LLMs or more-traditional models.

LLM-related work

Agents

FLAP: Flow-adhering planning with constrained decoding in LLMs
Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, Arshit Gupta

Attribute value extraction.png
Examples of attribute-value extraction from both textual and visual data. From "EIVEN: Efficient implicit attribute value extraction using multimodal LLM".

Attribute value extraction

EIVEN: Efficient implicit attribute value extraction using multimodal LLM
Henry Peng Zou, Gavin Yu, Ziwei Fan, Dan Bu, Han Liu, Peng Dai, Dongmei Jia, Cornelia Caragea

Continual learning

Q-Tuning: Queue-based prompt tuning for lifelong few-shot language learning
Yanhui Guo, Shaoyuan Xu, Jinmiao Fu, Jia (Kevin) Liu, Chaosheng Dong, Bryan Wang

Dialogue

Leveraging LLMs for dialogue quality measurement
Jinghan Jia, Abi Komma, Timothy Leffel, Xujun Peng, Ajay Nagesh, Tamer Soliman, Aram Galstyan, Anoop Kumar

Hallucination mitigation

Less is more for improving automatic evaluation of factual consistency
Tong Wang, Ninad Kulkarni, Yanjun (Jane) Qi

TofuEval: Evaluating hallucinations of LLMs on topic-focused dialogue summarization
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Justin Sun, Yi Zhang, Saab Mansour, Kathleen McKeown

Towards improved multi-source attribution for long-form answer generation
Nilay Patel, Shivashankar Subramanian, Siddhant Garg, Pratyay Banerjee, Amita Misra

Machine translation

A preference-driven paradigm for enhanced translation with large language models
Dawei Zhu, Sony Trenous, Xiaoyu Shen, Dietrich Klakow, Bill Byrne, Eva Hasler

Natural-language processing

Toward informal language processing: Knowledge of slang in large language models
Zhewei Sun, Qian Hu, Rahul Gupta, Richard Zemel, Yang Xu

Question answering

Bring your own KG: Self-supervised program synthesis for zero-shot KGQA
Dhruv Agarwal, Rajarshi (Raj) Das, Sopan Khosla, Rashmi Gangadharaiah

KG QA.png
The universal question-answering model presented in "Bring your own KG: Self-supervised program synthesis for zero-shot KGQA" uses a three-stage method to efficiently adapt to a new knowledge graph — without any training data.

Reasoning

CoMM: Collaborative multi-agent, multi-reasoning-path prompting for complex problem solving
Pei Chen, Boran Han, Shuai Zhang

Recommender systems

RecMind: Large language model powered agent for recommendation
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, Yingzhen Yang

Reinforcement learning from human feedback

RS-DPO: A hybrid rejection sampling and direct preference optimization method for alignment of large language models
Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, Prathap Ramachandra

Responsible AI

ITERALIGN: Iterative constitutional alignment of large language models
Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, Wei Wang

LLM alignment.png
In "ITERALIGN: Iterative constitutional alignment of large language models", Amazon researchers propose "an iterative method for aligning LLMs with human values and societal norms to ensure their reliability and safety."

MICo: Preventative detoxification of large language models through inhibition control
Roy Siegelmann, Ninareh Mehrabi, Palash Goyal, Prasoon Goyal, Lisa Bauer, Jwala Dhamala, Aram Galstyan, Rahul Gupta, Reza Ghanadan

The steerability of large language models toward data-driven personas
Junyi Li, Charith Peris, Ninareh Mehrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, Rahul Gupta

Retrieval-augmented generation

Enhancing contextual understanding in large language models through contrastive decoding
Zheng Zhao, Emilio Monti, Jens Lehmann, Haytham Assem

Text generation

Low-cost generation and evaluation of dictionary example sentences
Bill Cai, Clarence Ng, Daniel Tan, Shelvia Hotama

Multi-review fusion-in-context
Aviv Slobodkin, Ori Shapira, Ran Levy, Ido Dagan

Vision-language models

MAGID: An automated pipeline for generating synthetic multi-modal datasets
Hossein Aboutalebi, Justin Sun, Hwanjun Song, Yusheng Xie, Arshit Gupta, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour

Prompting vision-language models for aspect-controlled generation of referring expressions
Danfeng Guo, Sanchit Agarwal, Arpit Gupta, Jiun-Yu Kao, Emre Barut, Tagyoung Chung, Jing Huang, Mohit Bansal

Referring-expression generation.png
Examples of the tasks of image captioning, dense captioning, and referring-expression generation. From "Prompting vision-language models for aspect-controlled generation of referring expressions".

General and classical techniques

Conversational agents

Leveraging interesting facts to enhance user engagement with conversational interfaces
Nikhita Vedula, Giuseppe Castellucci, Eugene Agichtein, Oleg Rokhlenko, Shervin Malmasi

Information extraction

Leveraging customer feedback for multi-modal insight extraction
Sandeep Sricharan Mukku, Abinesh Kanagarajan, Pushpendu Ghosh, Chetan Aggarwal

REXEL: An end-to-end model for document-level relation extraction and entity linking
Nacime Bouziani, Shubhi Tyagi, Joseph Fisher, Jens Lehmann, Andrea Pierleoni

Machine learning

DEED: Dynamic early exit on decoder for accelerating encoder-decoder transformer models
Peng Tang, Pengkai Zhu, Tian Li, Srikar Appalaraju, Vijay Mahadevan, R. Manmatha

Machine translation

How lexical is bilingual lexicon induction?
Harsh Kohli, Helian Feng, Nicholas Dronen, Calvin McCarter, Sina Moeini, Ali Kebarighotbi

M3T: A new benchmark dataset for multi-modal document-level machine translation
Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fujinuma, Maria Nădejde, Xing Niu, Yair Kittenplon, Ron Litman, Raghavendra Pappagari

Responsible AI

Mitigating bias for question answering models by tracking bias influence
Mingyu Derek Ma, Jiun-Yu Kao, Arpit Gupta, Yu-Hsiang Lin, Wenbo Zhao, Tagyoung Chung, Wei Wang, Kai-Wei Chang, Nanyun Peng

Semantic retrieval

Extremely efficient online query encoding for dense retrieval
Nachshon Cohen, Yaron Fairstein, Guy Kushilevitz

Text summarization

CCSUM: A large-scale and high-quality dataset for abstractive news summarization
Xiang Jiang, Markus Dreyer

CCSum.png
The process for constructing the CCSum dataset. First, news articles are clustered into news events, from which candidate article-summary pairs are generated. Extensive filtering distills the candidates into the final dataset. From "CCSum: A large-scale and high-quality dataset for abstractive news summarization".

Semi-supervised dialogue abstractive summarization via high-quality pseudolabel selection
Jianfeng He, Hang Su, Jason Cai, Igor Shalyminov, Hwanjun Song, Saab Mansour

Visual question answering

Multiple-question multiple-answer text-VQA
Peng Tang, Srikar Appalaraju, R. Manmatha, Yusheng Xie, Vijay Mahadevan

Research areas
Tags
About the Author
Staff writer

Related content

Work with us

See more jobs See more jobs
Applied Scientist II, Physics Simulation, Amazon Industrial Robotics
US, MA, N.reading
Amazon Industrial Robotics is seeking exceptional talent to help develop the next generation of advanced robotics systems that will transform automation at Amazon's scale. We're building revolutionary robotic systems that combine cutting-edge AI, sophisticated control systems, and advanced mechanical design to create adaptable automation solutions capable of working safely alongside humans in dynamic environments. This is a unique opportunity to shape the future of robotics and automation at an unprecedented scale, working with world-class teams pushing the boundaries of what's possible in robotic dexterous manipulation, locomotion, and human-robot interaction. We are seeking a talented Applied Scientist to join our advanced robotics team, focusing on developing and applying cutting-edge simulation methodologies for advanced robotics systems. This role centers on research and development of physics-based simulation techniques, sim-to-real transfer methods, and machine learning approaches that enable rapid development, testing, and validation of robotic systems operating in complex, real-world environments. Key job responsibilities - Advance physics-based simulation fidelity for contact-rich manipulation and locomotion - Design and build high-performance simulation tools integrated into a production robotics stack - Translate research ideas into robust, scalable software pipelines - Develop methods to quantify and reduce simulation-to-reality gaps across design, safety, and control - Architect scalable simulation solutions for rigid and deformable body dynamics - Build simulation pipelines optimized for large-scale reinforcement and policy learning - Establish frameworks for continuous simulation improvement using real-world deployment data - Collaborate with engineering, science, and safety teams on simulation requirements and validation About the team Our team is building a comprehensive simulation platform for advanced robotics development, combining locomotion and manipulation capabilities. We operate at the cutting edge of physics simulation, reinforcement learning, and sim-to-real transfer, collaborating with world-class robotics engineers, applied scientists, and mechanical designers in a fast-paced, innovation-driven environment. This role uniquely combines fundamental research with real-world deployment. You will pursue core research questions in physics-based simulation while seeing your work translated into production systems, validated on real hardware, and informed by deployment data. Working alongside Simulation Software Engineers, you will help transform research ideas into scalable, production-grade simulation capabilities that directly impact how robots are designed, trained, and deployed.
Senior Satellite GNC Engineer, Amazon Leo
US, WA, Redmond
Amazon Leo is Amazon’s low Earth orbit satellite network. Our mission is to deliver fast, reliable internet connectivity to customers beyond the reach of existing networks. From individual households to schools, hospitals, businesses, and government agencies, Amazon Leo will serve people and organizations operating in locations without reliable connectivity. Export Control Requirement: Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. This position is part of the Satellite Attitude Determination and Control team. You will design and analyze the control system and algorithms, support development of our flight hardware and software, help integrate the satellite in our labs, participate in flight operations, and see a constellation of satellites flow through the production line into orbit. Key job responsibilities - Design and analyze algorithms for estimation, flight control, and precise pointing using linear methods and simulation. - Develop and apply models and simulations, with various levels of fidelity, of the satellite and our constellation. - Component level environmental testing, functional and performance checkout, subsystem integration, satellite integration, and in space operations. - Manage the spacecraft constellation as it grows and evolves. - Continuously improve our ability to serve customers by maximizing payload operations time. - Develop autonomy for Fault Detection and Isolation on board the spacecraft. A day in the life This is an opportunity to play a significant role in the design of an entirely new satellite system with challenging performance requirements. The large, integrated constellation brings opportunities for advanced capabilities that need investigation and development. The constellation size also puts emphasis on engineering excellence so our tools and methods, from conceptualization through manufacturing and all phases of test, will be state of the art as will the satellite and supporting infrastructure on the ground. You will find that Amazon Leo's mission is compelling, so our program is staffed with some of the top engineers in the industry. Our daily collaboration with other teams on the program brings constant opportunity for discovery, learning, and growth. About the team Our team has lots of experience with various satellite systems and many other flight vehicles. We have bench strength in both our mission and core GNC disciplines. We design, prototype, test, iterate and learn together. Because GNC is central to safe flight, we tend to drive Concepts of Operation and many system level analyses.
Applied Scientist, Amazon Business Ads
US, NY, New York
Advertising at Amazon is growing incredibly fast and we are responsible for defining and delivering a collection of advertising products that drive discovery and sales. Amazon Business Ads is equally growing fast ($XXXMs to $XBs) and owns engineering and science for the AB WW ad experience. We build business-to-business (“B2B”) specific ad solutions distributed across retail and ad systems for shopper and advertiser experiences. Some include new ad placements or widgets, creatives, sourcing techniques, ad campaign management capabilities and much more! We consider unique AB qualities which are differentiated from the consumer experience such as varying shopper role types, purchasing complexities based on business size and industry (eg education vs healthcare), AB specific features (eg business discounts, buying policies to restrict and prefer products), and AB buyer behaviors (eg buying in bulk). We are seeking a scientific leader who can drive innovation in complex problem areas and new business initiatives. The ideal candidate will: Technical & Research Requirements: * Demonstrate fluency in Python, R, Matlab or other statistical languages and familiarity with deep learning frameworks like PyTorch, TensorFlow * Lead end-to-end solution development from research to prototyping and experimentation * Write and deploy significant parts of scientifically novel software solutions into production Leadership & Influence: * Drive team's scientific agenda by proposing new initiatives and securing management buy-in including PM, SDM * Build consensus on large projects and influence decisions across different teams in Ads Key Leadership Principles: * Dive Deep: Uncover non-obvious insights in data * Deliver Results: Create solutions aligned with customer and product needs * Learn and Be Curious: Demonstrate self-driven desire to explore new research areas * Earn Trust: Build relationships with stakeholders through understanding business needs
Applied Scientist, Prime Video - Playback Intelligence
US, WA, Seattle
Prime Video is a first-stop entertainment destination offering customers a vast collection of premium programming in one app available across thousands of devices. Prime members can customize their viewing experience and find their favorite movies, series, documentaries, and live sports – including Amazon MGM Studios-produced series and movies; licensed fan favorites; and programming from Prime Video add-on subscriptions such as Apple TV+, Max, Crunchyroll and MGM+. All customers, regardless of whether they have a Prime membership or not, can rent or buy titles via the Prime Video Store, and can enjoy even more content for free with ads. Are you interested in shaping the future of entertainment? Prime Video's technology teams are creating best-in-class digital video experience. As a Prime Video technologist, you’ll have end-to-end ownership of the product, user experience, design, and technology required to deliver state-of-the-art experiences for our customers. You’ll get to work on projects that are fast-paced, challenging, and varied. You’ll also be able to experiment with new possibilities, take risks, and collaborate with remarkable people. We’ll look for you to bring your diverse perspectives, ideas, and skill-sets to make Prime Video even better for our customers. With global opportunities for talented technologists, you can decide where a career Prime Video Tech takes you! As an Applied Scientist in the Prime Video Playback Intelligence Organization, you will have deep subject matter expertise in applied machine learning and data science, with specializations in video streaming optimization, information retrieval, anomaly detection and root-causing systems, large language models and generative AI across various modalities. Key job responsibilities - Work with multiple teams of scientists, engineers, and product managers to translate business and functional requirements into concrete deliverables leading strategic efforts to enhance customer quality of experiences. - Work on problems spaces such as: improving the customer playback quality of experience across Video on Demand, Live Events and Linear Content. - Reduce the time/cost/effort to optimize the customer experience as well as detect, root-cause, and mitigate defects in the customer experience. You’ll seek to understand the depth and nuance of streaming video at scale and identify opportunities to grow our business and improve customer quality of experience via principled ML/AI solutions. - Lead integration of new algorithms and processes into existing modeling stacks, simplify and streamline the existing modeling stacks, and develop testing and evaluation strategies. Ultimately, you'll work backwards from the desired outcomes and lead the way on determining the ideal solution (statistical techniques, traditional ML, GenAI, etc). A day in the life We love solving challenging and hard problems in our quest to innovate on behalf of our customers and provide the best video streaming experience. We push the boundaries to leverage and invent technologies which help create unrivaled experiences for our customers to help us move fast in a growing and changing environment. We use data to guide our decisions, work closely with our engineering and product counterparts, and partner with other Science teams as well as academic institutions to learn and guide in an environment of innovation.
Applied Scientist, Selection Monitoring
IN, KA, Bengaluru
Selection Monitoring team is responsible for making the biggest catalog on the planet even bigger. In order to drive expansion of the Amazon catalog, we develop advanced ML/AI technologies to process billions of products and algorithmically find products not already sold on Amazon. We work with structured, semi-structured and Visually Rich Documents using deep learning, NLP and image processing. The role demands a high-performing and flexible candidate who can take responsibility for success of the system and drive solutions from research, prototype, design, coding and deployment. We are looking for Applied Scientists to tackle challenging problems in the areas of Information Extraction, Efficient crawling at internet scale, developing ML models for website comprehension and agents to take multi-step decisions. You should have depth and breadth of knowledge in text mining, information extraction from Visually Rich Documents, semi structured data (HTML) and advanced machine learning. You should also have programming and design skills to manipulate Semi-Structured and unstructured data and systems that work at internet scale. You will encounter many challenges, including: - Scale (build models to handle billions of pages), - Accuracy (requirements for precision and recall) - Speed (generate predictions for millions of new or changed pages with low latency) - Diversity (models need to work across different languages, market places and data sources) You will help us to - Build a scalable system which can algorithmically extract information from world wide web. - Intelligently cluster web pages, segment and classify regions, extract relevant information and structure the data available on semi-structured web. - Build systems that will use existing Knowledge Base to perform open information extraction at scale from visually rich documents. Key job responsibilities - Use AI, NLP and advances in LLMs/SLMs and agentic systems to create scalable solutions for business problems. - Efficiently Crawl web, Automate extraction of relevant information from large amounts of Visually Rich Documents and optimize key processes. - Design, develop, evaluate and deploy, innovative and highly scalable ML models, esp. leveraging latest advances in RL-based fine tuning methods like DPO, GRPO etc. - Work closely with software engineering teams to drive real-time model implementations. - Establish scalable, efficient, automated processes for large scale model development, model validation and model maintenance. - Lead projects and mentor other scientists, engineers in the use of ML techniques. - Publish innovation in research forums.
Applied Scientist II, IES LATECH ML
BR, SP, Sao Paulo
Do you like working on projects that are highly visible and are tied closely to Amazon’s growth? Are you seeking an environment where you can drive innovation leveraging the scalability and innovation with Amazon's AWS cloud services? The Amazon International Technology Team is hiring Applied Scientists to work in our Machine Learning team in Mexico City. The Intech team builds International extensions and new features of the Amazon.com web site for individual countries and creates systems to support Amazon operations. We have already worked in Germany, France, UK, India, China, Italy, Brazil and more. Key job responsibilities About you You want to make changes that help millions of customers. You don’t want to make something 10% better as a part of an enormous team. Rather, you want to innovate with a small community of passionate peers. You have experience in analytics, machine learning, LLMs and Agentic AI, and a desire to learn more about these subjects. You want a trusted role in strategy and product design. You put the customer first in your thinking. You have great problem solving skills. You research the latest data technologies and use them to help you innovate and keep costs low. You have great judgment and communication skills, and a history of delivering results. Your Responsibilities - Define and own complex machine learning solutions in the consumer space, including targeting, measurement, creative optimization, and multivariate testing. - Design, implement, and evolve Agentic AI systems that can autonomously perceive their environment, reason about context, and take actions across business workflows—while ensuring human-in-the-loop oversight for high-stakes decisions. - Influence the broader team's approach to integrating machine learning into business workflows. - Advise leadership, both tech and non-tech. - Support technical trade-offs between short-term needs and long-term goals.
Senior Applied Scientist, Alexa International
US, WA, Bellevue
Alexa International Science team is looking for a passionate, talented, and inventive Senior Applied Scientist to help build industry-leading technology with Large Language Models (LLMs) and multimodal systems, requiring strong deep learning and generative models knowledge. At this level, you will drive cross-team scientific strategy, influence partner teams, and deliver solutions that have broad impact across Alexa's international products and services. Key job responsibilities As a Senior Applied Scientist with the Alexa International team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art with LLMs, particularly delivering industry-leading scientific research and applied AI for multi-lingual applications — a challenging area for the industry globally. Your work will directly impact our global customers in the form of products and services that support Alexa+. You will leverage Amazon's heterogeneous data sources and large-scale computing resources to accelerate advances in text, speech, and vision domains. The ideal candidate possesses a solid understanding of machine learning, speech and/or natural language processing, modern LLM architectures, LLM evaluation & tooling, and a passion for pushing boundaries in this vast and quickly evolving field. They thrive in fast-paced environment, like to tackle complex challenges, excel at swiftly delivering impactful solutions while iterating based on user feedback, and are able to influence and align multiple teams around a shared scientific vision.
Intern - Economics, Private Brands - Intelligence
US, NY, New York
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to apply their structural econometrics skillsets to solve real world problems. The intern will work in the area of Amazon Private Brands and develop models to improve our product selection. Our PhD Economist Internship Program offers hands-on experience in applied economics, supported by mentorship, structured feedback, and professional development. Interns work on real business and research problems, building skills that prepare them for full-time economist roles at Amazon and beyond. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. About the team The Amazon Private Brands science advance team applies Machine Learning, Statistics and Econometrics/economics to solve high-impact business problems, develop prototypes for Amazon-scale science solutions, and optimize key business functions of Amazon Private Brands and other Amazon orgs. We are an interdisciplinary team, using science and technology and leveraging the strengths of engineers and scientists to build solutions for some of the toughest business problems at Amazon, covering areas such as pricing, discovery, negotiation, forecasting, supply chain and product selection/development.
Research Scientist, Special Projects
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
Member of Technical Staff - Applied Science , AGI Autonomy
US, CA, San Francisco
Amazon has launched a new research lab in San Francisco to develop foundational capabilities for useful AI agents. We’re enabling practical AI to make our customers more productive, empowered, and fulfilled. Our work leverages large vision language models (VLMs) with reinforcement learning (RL) and world modeling to solve perception, reasoning, and planning to build useful enterprise agents. Our lab is a small, talent-dense team with the resources and scale of Amazon. Each team in the lab has the autonomy to move fast and the long-term commitment to pursue high-risk, high-payoff research. We’re entering an exciting new era where agents can redefine what AI makes possible. Key job responsibilities You will contribute directly to AI agent development in an applied research role to improve the multi-model perception and visual-reasoning abilities of our agent. Daily responsibilities including model training, dataset design, and pre- and post-training optimization. You will be hired as a Member of Technical Staff.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our monthly newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social