Innovations from the 2018 Alexa Prize

March 4 marked the kickoff of the third Alexa Prize Socialbot Grand Challenge, in which university teams build socialbots capable of conversing on a wide range of topics and make them available to millions of Alexa customers through the invitation “Alexa, let’s chat”. Student teams may begin applying to the competition now, and in the next six weeks, the Alexa Prize team will make a series of roadshow appearances at tech hubs in the U.S. and Europe to meet with students and answer questions about the program.As the third Alexa Prize Socialbot Grand Challenge gears up, the Alexa science blog is reviewing some of the technical accomplishments from the second. An earlier post examined contributions by Amazon’s Alexa Prize team; this one examines innovations from the participating university teams.

The 2018 Alexa Prize featured eight student teams from four countries, each of which adopted distinctive approaches to some of the central technical questions in conversational AI. We survey those approaches in a paper we released late last year, and the teams themselves go into even greater detail in the papers they submitted to the latest Alexa Prize Proceedings. Here, we touch on just a few of the teams’ innovations.

Handling automatic speech recognition (ASR) errors

Conversations with socialbots can cover a wide range of topics and named entities, which makes automatic speech recognition (ASR) more difficult than it is during more task-oriented interactions with Alexa. Consequently, all the teams in the 2018 Alexa Prize competition built computational modules to handle ASR errors.

Several teams built systems that prompted customers for clarification if ASR confidence scores were too low, and several others retained alternative high-scoring ASR interpretations for later re-evaluation.

Gunrock, the team from the University of California, Davis, that won the 2018 challenge, built a system for correcting ASR errors that uses the double-metaphone algorithm, an algorithm that produces standardized representations of word pronunciations. When the ASR system assigned a word a low confidence score, Gunrock’s error correction module would apply the double-metaphone algorithm to the word and then search a database of metaphone-encoded pronunciations for a partial match. Those pronunciations are grouped according to conversation topic, which lets the module take advantage of contextual information.

So, for instance, the metaphone representation of the phrase “secure holiday” is SKRLT, which doesn’t occur in Gunrock’s database. But SKRLT is a substring of the metaphone representation APSKRLTS, which does occur in the database. So Gunrock’s system would correct SKRLT to APSKRLTS and return the corresponding English phrase: “obscure holidays”.

Knowledge graphs

Carrying on a conversation requires knowledge, and most teams chose to encode their socialbots’ knowledge in graphs. A graph is a mathematical object consisting of nodes, usually depicted as circles, and edges, usually depicted as line segments connecting nodes. In a knowledge graph, the nodes might represent objects, and the edges might represent relationships between them. Several teams populated their knowledge graphs with data from open sources such as DBPedia, Wikidata, Reddit, Twitter, and IMDB, and many of the teams built their graphs using the Neptune graph database service from Amazon Web Services.

knowledge-graph.png._CB453975899_.png
An example of a knowledge graph built with Amazon Web Services’ Neptune service

Alana, the team from Heriot-Watt University in Scotland and the third-place finisher in the 2018 challenge, used Neptune to build a knowledge graph that encodes all the information in the Wikidata knowledge base, plus some additional data from the DBpedia knowledge base. When the Alana socialbot identifies a named entity in a conversation, it begins a context-constrained exploration of the graph, assembling a subgraph of linked concepts.

If someone chatting with the Alana bot mentioned the movie E.T., for example, Alana’s linked-concept generator would follow the Wikidata link from E.T. to the entry for Drew Barrymore, who appeared in the film, but not to the entry for Sweden, which is the second country in which the movie was released. Then, once it has built up a database of linked concepts, the Alana socialbot selects one at random to serve as the basis for a conversational response.

Natural-language understanding (NLU) for open-domain dialogue

Amazon researchers provided the student teams with default modules for doing natural-language understanding (NLU), or extracting linguistic meaning from raw text, but most teams chose to supplement them or, in some cases, supplant them with systems tailored specifically to the demands of conversational AI. Student teams built their own modules to classify utterances according to intent, or the goal the speaker hopes to achieve, and dialogue act, such as asking for information or requesting clarification; to identify the topics of utterances; and to assess the sentiments expressed by particular choices of phrasing, among other things.

Most of the NLU literature focuses on relatively short, goal-directed utterances. But in conversations with socialbots, people will often speak in longer, more complex sentences. So Gunrock built an NLU module that splits longer sentences into smaller, semantically distinct units, which then pass to additional NLU modules.

To train the segmentation module, Gunrock used movie-dialogue data from the Cornell Movie-Quotes Corpus, which had been annotated with a special tag (“<BRK>”) to indicate breaks between semantically distinct units. On a test set, the module was 95.25% accurate, and an informal review indicated that it was accurately segmenting customers’ remarks. For example, the raw ASR output “Alexa that is cool what do you think of the Avengers” was segmented into “Alexa <BRK> that is cool <BRK> what do you think of the Avengers <BRK>”.

Dialogue management

The outputs of the NLU modules, along with any other utterance data the teams deem useful, pass to the dialogue management module, which generates an array of possible responses and selects one to send to Alexa’s voice synthesizer.

Alquist, the team from the Czech Technical University in Prague and the runner-up in the 2018 challenge, used a hybrid code network (HCN) for its dialogue manager. An HCN combines a neural network with handwritten code that reflects the developers’ understanding of the problem space. HCNs can dramatically reduce the amount of training data required to achieve a given level of performance, by sparing the network from having to learn how to perform tasks that are easily coded.

In Alquist’s case, the added code has two main functions: it filters out suggested responses that violate a set of handwritten rules about what types of responses should follow what types of utterances, and it inserts context-specific data into responses selected by the neural net. So, for instance, the neural network might output the response “That movie was directed by {say_director}”, where {say director} is an instruction to a complementary program that has separately processed data from the NLU modules.

Customer experience and personalization

All of the teams had to address the question of when to switch conversation topics and how to select new topics, but Iris, the team from Emory University, built a machine learning model that predicted appealing topics on the basis of conversational history — what topics a customer had previously accepted and rejected and what types of interactions he or she had previously engaged in. Iris trained their model on data from their socialbot’s past interactions with customers.

In tests, Iris compared their model to the simple heuristic of suggesting new topics in order of overall popularity and found that, on average, their model’s recommendations were 62% more likely to be accepted.

It was a pleasure to work with the student teams who competed in the 2018 Alexa Prize and a privilege to witness their innovative approaches to a fundamental problem in artificial-intelligence research. We can’t wait to see what the next group of teams will come up with!

Latest news

The latest updates, stories, and more about Alexa Prize.
IL, Tel Aviv
Are you a MS or PhD student interested in a 2024 Research Science Internship, where you would be using your experience to initiate the design, development, execution and implementation of scientific research projects? If so, we want to hear from you! Is your research in machine learning, deep learning, automated reasoning, speech, robotics, computer vision, optimization, or quantum computing? If so, we want to hear from you! We are looking for motivated students with research interests in a variety of science domains to build state-of-the-art solutions for never before solved problems You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science Key job responsibilities As a Research Science Intern, you will have following key job responsibilities; • Work closely with scientists and engineering teams (position-dependent) • Work on an interdisciplinary team on customer-obsessed research • Design new algorithms, models, or other technical solutions • Experience Amazon's customer-focused culture A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships and up to 12 months for part time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
US, VA, Arlington
Amazon Web Services (AWS) is seeking a highly skilled Economist to help shape the future of our company and enhance the success of our customers. With AWS generating approximately $100B in annual revenue, we are expanding rapidly and need to identify the interventions that are most effective in helping both existing and potential customers throughout their cloud- adoption journey. As part of this role, you will apply advanced econometrics and machine learning techniques to determine which interventions yield the best outcomes across different stages of the customer journey, from early engagement to mature customer relationships. Your work will center on applying causal inference and machine learning to large, complex datasets, uncovering actionable insights that directly influence AWS's strategic decisions. You will be instrumental in developing scalable models that deepen our understanding of customer behavior and quantify the impact of marketing and sales initiatives. By working closely with key business stakeholders, you’ll ensure that AWS consistently delivers the most effective solutions tailored to the unique needs of our diverse and growing customer base. Key job responsibilities Key job responsibilities -Apply your expertise in econometrics and machine learning to evaluate the effectiveness of AWS interventions and customer engagement strategies. -Identify patterns and opportunities in customer data to suggest new interventions, such as credit offers, discounts, and service recommendations. -Formalize and document research processes, ensuring scientific rigor and knowledge sharing within Amazon’s science community. -Communicate insights and findings effectively to business leaders across various levels of the organization, influencing strategic decision-making.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics
LU, Luxembourg
Are you a MS or PhD student interested in a 2025 Internship in the field of machine learning, deep learning, speech, robotics, computer vision, optimization, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
LU, Luxembourg
At Global Mile Expansion team, our vision is to become the carrier of choice for all of our Selling Partners cross-border shipping needs, offering complete set of end to end cross border solutions from key manufacturing hubs to footprint countries supporting business who use Amazon to grow their business globally. As we expand, the need for comprehensive business insight and robust demand forecasting to aid decision making on asset utilization especially where we know demand will be variable becomes vital, as well as operational excellence. We are building business models involving large amounts of data and Macro economic inputs to produce the robust forecast to help the operational excellence and continue improving the customer experience. We are looking for an experienced economist who can apply innovative modelling techniques to real-world problems, and convert it to highly business-impacting solutions. Key job responsibilities - Experienced in using mathematical and statistical approach to create new, scalable solutions for business problems - Analyze and extract relevant information from business data to help automate and optimize key processes - Design, develop and evaluate highly innovative models for predictive learning - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Research and implement statistical approaches to understand the business long-term and short-term trend and support the strategies
ES, Madrid
Are you a MS or PhD student interested in a 2025 Internship in the field of machine learning, deep learning, speech, robotics, computer vision, optimization, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
US, WA, Seattle
When customers search for products on the Amazon website, they often see brand advertisements displayed right below the search bar. These ads are part of the Sponsored Brands (SB) program. Our team, the SB Search and Relevance team, works on solving challenges to retrieve the most relevant ads for a customer's search query. A customer's search query is typically a short, free-form text consisting of just a few words. Our algorithm needs to understand the customer's underlying intention from this limited information. At the same time, each advertisement consists of various elements like text descriptions, images, videos, and more. Our algorithm also needs to comprehend the content of these ads and identify the most relevant one from the large pool of ad candidates. As Amazon's advertising business is growing rapidly, we are looking for experienced applied scientists. As an Applied Scientist on this team, you will: - Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. - Apply deep learning and natural language processing to improve information retrieval and relevance. - Design and run A/B experiments. Evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Optimize deep learning inference latency by utilizing methods like knowledge distillation. - Work with software development engineers and write code to bring models into production. - Recruit Applied Scientists to the team and provide mentorship. Impact and Career Growth - You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! - Define a long-term science vision for our advertising business, driven fundamentally from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.
US, MA, Boston
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? Machine learning (ML) has been strategic to Amazon from the early years. We are pioneers in areas such as recommendation engines, product search, eCommerce fraud detection, and large-scale optimization of fulfillment center operations. The AWS Industries Team at AWS helps AWS customers implement Generative AI solutions and realize transformational business opportunities for AWS customers in the most strategic industry verticals. This is a team of data scientists, engineers, and architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and build applications to launch these solutions at scale. The AWS Industries team provides guidance and implements best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for Applied Scientists capable of using GenAI and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. Key job responsibilities As an Applied Scientist, you will- - Collaborate with AI/ML scientists, engineers, and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms and build ML systems to address real-world challenges - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production - Create and deliver best practice recommendations, tutorials, blog posts, publications, sample code, and presentations adapted to technical, business, and executive stakeholder. Publish novel developments in internal and external papers, forums, and conferences - Provide customer and market feedback to Product and Engineering teams to help define product direction About the team ABOUT AWS: Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, CA, San Diego
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The AWS Industries Team at AWS helps AWS customers implement Generative AI solutions and realize transformational business opportunities for AWS customers in the most strategic industry verticals. This is a team of data scientists, engineers, and architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and build applications to launch these solutions at scale. The AWS Industries team provides guidance and implements best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. In this Data Scientist role you will be capable of using GenAI and other techniques to design, evangelize, and implement and scale cutting-edge solutions for never-before-solved problems. Key job responsibilities As a Senior Data Scientist, you will- - Collaborate with AI/ML scientists, engineers, and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms and build ML systems to address real-world challenges - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production - Create and deliver best practice recommendations, tutorials, blog posts, publications, sample code, and presentations adapted to technical, business, and executive stakeholder - Provide customer and market feedback to Product and Engineering teams to help define product direction About the team ABOUT AWS: Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, WA, Bellevue
Amazon Last Mile builds global solutions that enable Amazon to attract an elastic supply of drivers, companies, and assets needed to deliver Amazon's and other shippers' volumes at the lowest cost and with the best customer delivery experience. Last Mile Science team owns the core decision models in the space of jurisdiction planning, delivery channel and modes network design, capacity planning for on the road and at delivery stations, routing inputs estimation and optimization. We also own scalable solutions to reduce risks, improve safety, enhance personalized experiences of our delivery associates and partners. Our research has direct impact on customer experience, driver and station associate experience, Delivery Service Partner (DSP)’s success and the sustainable growth of Amazon. We are looking for a passionate individual with strong machine learning and analytical skills to join its Last Mile Science team in the endeavor of designing and improving the most complex planning of delivery network in the world. As a Senior Data Scientist, you will work with software engineers, product managers, and business teams to understand the business problems and requirements, distill that understanding to crisply define the problem, and design and develop innovative solutions to address them. Our team is highly cross-functional and employs a wide array of scientific tools and techniques to solve key challenges, including supervised and unsupervised machine learning, non-convex optimization, causal inference, natural language processing, linear programming, reinforcement learning, and other forecast algorithms. Key job responsibilities Key job responsibilities * Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale and complexity. * Build Machine Learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. * Run A/B experiments, gather data, and perform statistical analysis. * Measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs. * Research new and innovative machine learning approaches. Help coach/mentor junior scientists in the team. * Willingness to publish research at internal and external top scientific venues. Write and pursue IP submissions.