Code and Datasets

MT-GenEval

Anna Currey, Maria Nădejde, Raghavendra Pappagari, Mia Mayer, Stanislas LAULY, Xing Niu, Benjamin Hsu, Georgiana Dinu

2022

Last updated May 7, 2024

Access

Share

Share

Share

Share

Access

Share

Share

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased (Freitag et al., 2021; Isabelle et al., 2017). In particular, gender accuracy in translation (Choubey et al., 2021; Saunders and Byrne, 2020) can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MTGenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.

Types

Text

Anomaly detection for graph-based data

Huijun (Lona) Yu

March 14, 2024

Diffusion modeling within the representational space of a variational autoencoder enables state-of-the-art results.

Machine learning
New tool, dataset help detect hallucinations in large language models

Xiangkun Hu, Dongyu Ru

January 17, 2024

Representing facts using knowledge triplets rather than natural language enables finer-grained judgments.

Conversational AI
Making automated visual-inspection systems practical

Tryambak Gangopadhyay

September 05, 2023

Benchmarking framework that includes a product-agnostic public dataset, guidelines for model selection, and an evaluation approach helps bridge the gap between research and real-world implementation.

Computer vision

Amazon Robotics - Applied Scientist I Intern / Co-op - 2025

US, MA, Westborough

Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics

Amazon Robotics - Applied Scientist I Intern / Co-op - 2025

US, MA, North Reading

Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics

Research Scientist, AIGC HR Analytics

US, NY, New York

The PXT (People Experience and Technology) Analytics team for AIGC (Ads, IMDb and Grand Challenge) is seeking a highly skilled and motivated Research Scientist to join our team. You will be an integral part of the Research Science space to support the AIGC PXT org initiatives. If you enjoy innovating, thinking big and want to contribute directly to the success of a growing team, you may be a prime candidate for this position. Key job responsibilities Design experiments, test hypotheses, and build actionable models. Conduct quantitative analyses of talent management data and trends. Conduct qualitative data collection and analysis. Partner closely and drive effective collaborations across multi-disciplinary research and product teams. Consult on appropriate analytic methodologies and scope research requests. Write comprehensive reports that summarize research methodology, results, and insights for both business and technical audiences.

Senior Applied Scientist, Amazon Q for Business Science

US, NY, New York

An AS III will lead complex projects in the GenAI space, specifically related to LLM-backed conversational agents that interact with multiple corporate data sources. The team works on RAG; QA from very rich documents, containing tables, plots, graphs, etc., multimodal documents, datatabase etc.; orchestration and planning multi-step actions; RAI aspects such as hallucination reduction and protection from attacks; and more. About the team Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Utility Computing (UC) AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (IoT), Platform, and Productivity Apps services in AWS, including support for customers who require specialized security solutions for their cloud services. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.

Applied Scientist, Generative Artificial Intelligence (AI) Innovation Center

AU, NSW, Sydney

AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest growing small- and mid-market accounts to enterprise-level customers including public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to customer success. AWS Support also partners with a global list of customers that are building mission-critical applications on top of AWS services. The Generative Artificial Intelligence (AI) Innovation Center team at AWS provides opportunities to innovate in a fast-paced organization that contributes to game-changing projects and technologies leveraging cutting-edge generative AI algorithms. As an Applied Scientist, you'll partner with technology and business teams to build solutions that surprise and delight our customers. We’re looking for Applied Scientists capable of using generative AI and other ML techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. Key job responsibilities - Collaborate with scientists and engineers to research, design and develop cutting-edge generative AI algorithms to address real-world challenges - Work across customer engagement to understand what adoption patterns for generative AI are working and rapidly share them across teams and leadership - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths for generative AI - Create and deliver best practice recommendations, tutorials, blog posts, sample code, and presentations adapted to technical, business, and executive stakeholder - Provide customer and market feedback to Product and Engineering teams to help define product direction. A day in the life Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. What if I don’t meet all the requirements? That’s okay! We hire people who have a passion for learning and are curious. You will be supported in your career development here at AWS. You will have plenty of opportunities to build your technical, leadership, business and consulting skills. Your onboarding will set you up for success, including a combination of formal and informal training. You’ll also have a chance to gain AWS certifications and access mentorship programs. You will learn from and collaborate with some of the brightest technical minds in the industry today.

Applied Scientist, Conversational AI ModEling and Learning

US, WA, Bellevue

Conversational AI ModEling and Learning (CAMEL) team is part of Amazon Devices organization where our mission is to build a best-in-class Conversational AI that is intuitive, intelligent, and responsive, by developing superior Large Language Models (LLM) solutions and services which increase the capabilities built into the model and which enable utilizing thousands of APIs and external knowledge sources to provide the best experience for each request across millions of customers and endpoints. We are looking for a passionate, talented, and resourceful Applied Scientist in the field of LLM, Artificial Intelligence (AI), Natural Language Processing (NLP), Recommender Systems and/or Information Retrieval, to invent and build scalable solutions for a state-of-the-art context-aware conversational AI. A successful candidate will have strong machine learning background and a desire to push the envelope in one or more of the above areas. The ideal candidate would also have hands-on experiences in building Generative AI solutions with LLMs, enjoy operating in dynamic environments, be self-motivated to take on challenging problems to deliver big customer impact, moving fast to ship solutions and then iterating on user feedback and interactions. Key job responsibilities As an Applied Scientist, you will leverage your technical expertise and experience to collaborate with other talented applied scientists and engineers to research and develop novel algorithms and modeling techniques to reduce friction and enable natural and contextual conversations. You will analyze, understand and improve user experiences by leveraging Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in artificial intelligence. You will work on core LLM technologies, including Prompt Engineering and Optimization, Supervised Fine-Tuning, Learning from Human Feedback, Evaluation, Self-Learning, etc. Your work will directly impact our customers in the form of novel products and services.

Sr. Applied Scientist, Private Brands Discovery

US, WA, Seattle

The Private Brands Discovery team designs innovative machine learning solutions to drive customer awareness for Amazon’s own brands and help customers discover products they love. Private Brands Discovery is an interdisciplinary team of Scientists and Engineers, who incubate and build disruptive solutions using cutting-edge technology to solve some of the toughest science problems at Amazon. To this end, the team employs methods from Natural Language Processing, Deep learning, multi-armed bandits and reinforcement learning, Bayesian Optimization, causal and statistical inference, and econometrics to drive discovery across the customer journey. Our solutions are crucial for the success of Amazon’s own brands and serve as a beacon for discovery solutions across Amazon. This is a high visibility opportunity for someone who wants to have business impact, dive deep into large-scale problems, enable measurable actions on the consumer economy, and work closely with scientists and engineers. As a scientist, you bring business and industry context to science and technology decisions. You set the standard for scientific excellence and make decisions that affect the way we build and integrate algorithms. Your solutions are exemplary in terms of algorithm design, clarity, model structure, efficiency, and extensibility. You tackle intrinsically hard problems, acquiring expertise as needed. You decompose complex problems into straightforward solutions.. With a focus on bias for action, this individual will be able to work equally well with Science, Engineering, Economics and business teams. Key job responsibilities - Drive applied science projects in machine learning end-to-end: from ideation over prototyping to launch. For example, starting from deep scientific thinking about new ways to support customers’ journeys through discovery, you analyze how customers discover, review and purchase Private Brands to innovate marketing and merchandising strategies. - Propose viable ideas to advance models and algorithms, with supporting argument, experiment, and eventually preliminary results. - Invent ways to overcome technical limitations and enable new forms of analyses to drive key technical and business decisions. - Present results, reports, and data insights to both technical and business leadership. - Constructively critique peer research and mentor junior scientists and engineers. - Innovate and contribute to Amazon’s science community and external research communities.

Applied Scientist, Conversational Ad Experiences - Sponsored Products & Brands

US, NY, New York

Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities. This position will be part of the Conversational Ad Experiences team within the Amazon Advertising organization. Our cross-functional team focuses on designing, developing and launching innovative ad experiences delivered to shoppers in conversational contexts. We utilize leading-edge engineering and science technologies in generative AI to help shoppers discover new products and brands through intuitive, conversational, multi-turn interfaces. We also empower advertisers to reach shoppers, using their own voice to explain and demonstrate how their products meet shoppers' needs. We collaborate with various teams across multiple Amazon organizations to push the boundary of what's possible in these fields. We are seeking an applied scientist for our team within the Sponsored Products & Brands organization. You'll be working with talented scientists, engineers, and product managers to innovate on behalf of our customers. An ideal candidate is able to navigate through ambiguous requirements, working with various partner teams, and has experience in generative AI, large language models (LLMs), information retrieval, and recommendation systems. Using a combination of generative AI and online experimentation, our scientists develop insights and optimizations that enable the monetization of Amazon properties while enhancing the experience of hundreds of millions of Amazon shoppers worldwide. If you're fired up about being part of a dynamic, driven team, then this is your moment to join us on this exciting journey! Key job responsibilities - Drive end-to-end generative AI projects that have a high degree of ambiguity, scale, and complexity. - Perform hands-on analysis of data sets to identify insights and build models that enhance traffic monetization, merchandise sales, and the overall shopper experience. - Train generative AI and machine learning models, run proof-of-concept experiments, optimize, and deploy models at scale in production. - Work closely with software engineers on detailed requirements, technical designs and implementation of end-to-end solutions in production. - Research new and innovative generative AI and machine learning approaches. - Work closely with product managers to contribute to our mission, and proactively identify opportunities where cutting edge science can help improve customer experience. - Be a member of the Amazon-wide machine learning community, participating in internal and external meetups, hackathons and conferences.

Senior Data Scientist, SCOT Inventory Placement

US, WA, Bellevue

Come and be a part of Amazon's amazing growth story! If you are looking for an opportunity to solve deep technical problems and build innovative solutions in a fast paced environment working with smart, passionate team members, this might be the role for you! Amazon Supply Chain Optimization Technology (SCOT) powers Amazon’s fulfillment network, determining how much of a given product is needed at locations around the world in order to ensure product availability while maintaining optimal inventory levels at each storage location. This includes strategically placing vendor orders and moving inventory across our network to serve customer demand as quickly as possible at reduced cost. (Learn more about SCOT: http://bit.ly/amazon-scot) The Inventory Placement team is seeking for Sr Data Scientist with strong analytical and communication skills to join our team. As a Senior Data Scientist for Inventory Placement, you will be responsible for driving improvements in how our inventory is distributed, stored, and replenished across our supply chain network. Your expertise in machine learning, statistical modeling, and optimization will enable our system teams to make smarter, data-backed decisions to finally ensure that products are placed in the right locations, at the right time, and in the right quantities. You will work closely with cross-functional teams in supply chain, engineering, and product to create scalable solutions that improve inventory efficiency and reduce operational costs. Key job responsibilities - Analysis of large amounts of data from different parts of the supply chain and their associated business functions using SQL. - Improving upon existing machine learning methodologies by developing new data sources, developing and testing model enhancements, running computational experiments, and fine-tuning model parameters for new models - Measure the impact of new features in our systems to own data driven decisions for future investments. - Formalizing assumptions about how models are expected to behave, creating definitions of outliers, developing methods to systematically identify these outliers, and explaining why they are reasonable or identifying fixes for them - Communicating verbally and in writing to business customers with various levels of technical knowledge, educating them about our research, as well as sharing insights and recommendations - Utilizing code (SQL, Python, R, etc.) for analyzing data and building statistical and machine learning models and algorithms About the team Our Inventory Placement team owns the systems that decide where in Amazon's fulfillment network to put inventory for the millions of products that Amazon sells. We build optimization models to make these placement decisions driven by signals such as forecasted customer demand, the cost and speed of shipping from each warehouse to each customer, and the available capacity at various points in our network. Inventory placement is central to achieving Amazon's objectives to minimize the cost of fulfillment while offering selection to customers at the fastest possible delivery speeds. Our systems are built entirely in-house, and are on the cutting edge in automated large scale supply chain planning and optimization systems. We're simultaneously developing the science of supply chain planning and solving some of the toughest computational challenges at Amazon.

Applied Scientist III, 5365 - CMT Systems

IN, KA, Bengaluru

Applied Scientist, CMT, Amazon Bangalore Impact As a member of the CMT team, you'll play a key role in the evolution of our Competitive Monitoring systems to solve significantly complex and interesting technical challenges in Large-scale computing, Distributed systems, Web applications, Data mining, Scalability, Security, and Algorithms to name a few. The team's work directly impacts customer experience at a worldwide scale. Innovation Are you seeking an environment where you can drive innovation? Do you want to apply state-of-the-art computer science and advance information retrieval techniques to solve real world problems of competitive data analysis? Does the challenge of building real time, highly scalable solutions for the most complex online business using innovative technology excite you? Opportunity To meet these challenges, the CMT team is looking for passionate, talented and innovative scientists looking to work on cutting edge technology, from Natural Language processing to optimization to image processing and LLMs. In addition to getting the opportunity to participate in research in several domains, you will lead the solution to complex pricing problems in an extremely agile environment. This role will have the opportunity to learn and work on the most cutting edge generative AI solutions. Come be part of this growing, dynamic and challenging space! Key job responsibilities 1. Research the problem domain and come up with various approaches to solve the problem. 2. Be willing to experiment quickly and fail fast. 3. Collaborate with engineers to come up with the right end to end solution to the business problems. 4. Ideate on future roadmap for science in CMT, and CMT in general.