Vancouver, Canada

3 important themes from Amazon's 2019 NeurIPS papers

Time series forecasting, bandit problems, and optimization are integral to Amazon's efforts to deliver better value for its customers.

Last year, the first 2,000-2,500 publicly released tickets to the Conference on Neural Information Processing Systems, or NeurIPS, sold out in 12 minutes.

This year, the conference organizers moved to a lottery system, allowing aspiring attendees to register in advance and randomly selecting invitees from the pool of registrants. But they also bumped the number of public-release tickets up from around 2,000 to 3,500, testifying to the conference’s continued popularity.

At NeurIPS this year, there are 26 papers with Amazon coauthors. They cover a wide range of topics, but surveying their titles, Alex Smola, a vice president and distinguished scientist in the Amazon Web Services organization, discerns three prominent themes, all tied to Amazon’s efforts to deliver better value for its customers.

Those three themes are time series forecasting (and causality), bandit problems, and optimization.

1. Time series forecasting

Time series forecasting involves measuring some quantity over time — such as the number of deliveries in a particular region in the past six months, or the number of cloud servers required to support a particular site over the past two years — and attempting to project that quantity into the future.

“That’s something that is very dear to Amazon’s heart,” Smola says. “For anything that Amazon does, it’s really beneficial to have a good estimate of what our customers will expect from us ahead of time. Only by being able to do that will we be able to satisfy customers’ demands, be it for products or services.”

A sequence of basis time series, forecast into the near future and summed together to approximate a new time series.
The paper “Think Globally, Act Locally” examines data sets with many correlated time series, such as the demand curves for millions of products sold online. The researchers describe a method for constructing a much smaller set of “basis time series”; the time series for any given product can be approximated by a weighted sum of the bases.
Courtesy of the researchers

The basic mathematical framework for time series forecasting is a century old, but the scale of modern forecasting problems calls for new analytic techniques, Smola says.

“Problems are nowadays highly multivariate,” Smola says. “If you look at the many millions of products that we offer, you want to be able to predict fairly well what will sell, where and to whom.

“You need to make reasonable assumptions on how this very large problem can be decomposed into smaller, more tractable pieces. You make structural approximations, and sometimes those structural approximations are what leads to very different algorithms.

“So you might, for instance, have a global model, and then you have local models that address the specific items or address the specific sales. If you look at ‘Think Globally, Act Locally’” — a NeurIPS paper whose first author is Rajat Sen, an applied scientist in the Amazon Search group — “it’s already in the title. Or look at ‘High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes’. In this case, you have a global structure, but it’s only in a small subspace where interesting things happen.”

Side-by-side images depict correlations between taxi traffic at different points in Manhattan at different times of day
The paper "High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes" describes a method for predicting correlations among many parallel time series. In one example, the researchers forecast correlations between the taxi traffic at different points in New York City at different times of day. Red lines indicate strong correlations; blue lines indicate strong negative correlations. Weekend midday traffic patterns (left) show negative correlations between locations near the Empire State Building, suggesting that taxis tend to prefer different routes depending on traffic conditions. Weekend evening traffic patterns show positive correlations between the vicinity of the Empire State Building and areas with high concentrations of hotels.
Courtesy of the researchers

An aspect of forecasting that has recently been drawing more attention, Smola says, is causality. Where traditional machine learning models merely infer statistical correlations between data points, “it is ultimately the causal relationship that matters,” Smola says.

“I think that causality is one of the most interesting conceptual developments affecting modern machine learning,” says Bernhard Schölkopf, like Smola a vice president and distinguished scientist in Amazon Web Services. “This is the main topic that I have been interested in for the last decade.”

Two of Schölkopf’s NeurIPS papers — “Perceiving the Arrow of Time in Autoregressive Motion” and “Selecting Causal Brain Features with a Single Conditional Independence Test per Feature” — address questions of causality, as does “Causal Regularization”, a paper by Dominik Janzing, a senior research scientist in Smola’s group.

“Normal machine learning builds on correlations of other statistical dependences,” Schölkopf explains. “This is fine as long as the source of the data doesn't change. For example, if in the training set of an image recognition system, all cows are standing on green pasture, then it is fine for an ML system to use the green as a useful feature in recognizing cows, as long as the test set looks the same. If in the test set, the cows are standing on the beach, then such a purely statistical system can fail.

“More generally: causal learning and inference attempts to understand how systems respond to interventions and other changes, and not just how to predict data that looks more or less the same as the training data.”

2. Bandit problems

The second major theme that Smola discerns in Amazon scientists’ NeurIPS papers is a concern with bandit problems, a phrase that shows up in the titles of Amazon papers such as “MaxGap Bandit: Adaptive Algorithms for Approximate Ranking” and “Low-Rank Bandit Methods for High-Dimensional Dynamic Pricing”. Bandit problems take their name from one-armed bandits, or slot machines.

“It used to be that those bandits were all mechanical, so there would be slight variations between them, and some would have maybe a slightly a higher return than others,” Smola explains. “I walk into a den of iniquity, and I want to find the one-armed bandit where I will lose the least money or maybe make some money. And the only feedback I have is that I pull arms, and I get money or lose money. These are very unreliable, noisy events.”

Bandit problems present what’s known as an explore-exploit trade-off. The gambler must simultaneously explore the environment — determine which machines pay out the most — and exploit the resulting knowledge — concentrate as much money as possible on the high-return machines. Early work on bandit problems concerned identifying the high-return machines with minimal outlays.

“That problem was solved about 20 years ago,” Smola says. “What hasn’t been solved — and this is where things get a lot more interesting — is once you start adding context. Imagine that I get to show you various results as you’re searching for your next ugly Christmas sweater. The unfortunate thing is that the creativity of sweater designers is larger than what you can fit on a page. Now the context is essentially, what time, where from, which user, all those things. We want to find and recommend the ugly Christmas sweater that works specifically for you. This is an example where context is immediately relevant.”

It’s really beneficial to have a good estimate of what our customers will expect from us ahead of time. Only by being able to do that will we be able to satisfy customers’ demands.
Alex Smola, VP and distinguished scientist, Amazon

In the bandit-problem framework, in other words, the high-payout machines change with every new interaction. But there may be external signals that indicate how they’re changing.

Distributed computing, which is inescapable for today’s large websites, changes the structure of the bandit problem, too.

“Say you go to a restaurant, and the cook wants to improve the menu,” Smola says. “You can try out lots of new menu items, and that’s a good way to improve the menu overall. But if you start offering a lot of undercooked dishes because you’re experimenting, then at some point your loyal customers will stay away.

“Now imagine you have 100 restaurants, and they all do the same thing at the same time. They can’t necessarily communicate at the per-second level; maybe every day or every week they chat with each other. Now this entire exploration problem becomes a little more challenging, because if two restaurants try out the same undercooked dish, you make the customer less happy than you could have.

“So how does this map back into Amazon land? Well, if you have many servers doing this recommendation, the explore-exploit trade-off might be too aggressive if every one of them works on their own.”

3. Optimization

Finally, Smola says, “There is a third category of results that has to do with making algorithms faster. If you look at ‘Primal-Dual Block Frank-Wolfe’, ‘Communication-Efficient Distributed SGD with Sketching’, ‘Qsparse-Local-SGD’ — those are the workhorses that run underneath all of this. Making them more efficient is obviously something that we care about, so we can respond to customer requests faster, train algorithms faster.”

Bird’s-eye view

NeurIPS is a huge conference, with more than 1,400 accepted papers that cover a bewildering variety of topics. Beyond the Amazon papers, Caltech professor and Amazon fellow Pietro Perona identifies three research areas as growing in popularity.

“One is understanding how deep networks work, so that we can better design architectures and optimization algorithms to train models,” Perona says. “Another is low-shot learning. Machines are still much less efficient than humans at learning, in that they need more training examples to achieve the same performance. And finally, AI and society — identifying opportunities for social good, sustainable development, and the like.”

NeurIPS is being held this year at the Vancouver Convention Center, and the main conference runs from Dec. 8 to Dec. 12. The Women in Machine Learning Workshop, for which Amazon is a gold-level sponsor, takes place on Dec. 9; the Third Conversational AI workshop, whose organizers include Alexa AI principal scientist Dilek Hakkani-Tür, will be held on Dec. 14.

Amazon's involvement at NeurIPS

Paper and presentation schedule

Tuesday, 12/10 | 10:45-12:45pm | East Exhibition Hall B&C

A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning | #192
Francisco Garcia (UMass Amherst/Amazon) · Philip Thomas (UMass Amherst)

Blocking Bandits | #17
Soumya Basu (UT Austin) · Rajat Sen (UT Austin/Amazon) · Sujay Sanghavi (UT Austin/Amazon) · Sanjay Shakkottai (UT Austin)

Causal Regularization | #180
Dominik Janzing (Amazon)

Communication-Efficient Distributed SGD with Sketching | #81
Nikita Ivkin (Amazon) · Daniel Rothchild (University of California, Berkeley) · Md Enayat Ullah (Johns Hopkins University) · Vladimir Braverman (Johns Hopkins University) · Ion Stoica (UC Berkeley) · Raman Arora (Johns Hopkins University)

Learning Distributions Generated by One-Layer ReLU Networks | #49
Shanshan Wu (UT Austin) ·Alexandros G. Dimakis (UT Austin) · Sujay Sanghavi (UT Austin/Amazon)

Tuesday, 12/10 | 5:30-7:30pm | East Exhibition Hall B&C

Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control | #195
Sai Qian Zhang (Harvard University) · Qi Zhang (Amazon) · Jieyu Lin (University of Toronto)

Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products | #37
Tharun Kumar Reddy Medini (Rice University) · Qixuan Huang (Rice University) · Yiqiu Wang (Massachusetts Institute of Technology) · Vijai Mohan (Amazon) · Anshumali Shrivastava (Rice University/Amazon)

Iterative Least Trimmed Squares for Mixed Linear Regression | #50
Yanyao Shen (UT Austin) · Sujay Sanghavi (UT Austin/Amazon)

Meta-Surrogate Benchmarking for Hyperparameter Optimization | #6
Aaron Klein (Amazon) · Zhenwen Dai (Spotify) · Frank Hutter (University of Freiburg) · Neil Lawrence (University of Cambridge) · Javier Gonzalez (Amazon)

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations | #32
Debraj Basu (Adobe) · Deepesh Data (UCLA) · Can Karakus (Amazon) · Suhas Diggavi (UCLA)

Selecting Causal Brain Features with a Single Conditional Independence Test per Feature | #139
Atalanti Mastakouri (Max Planck Institute for Intelligent Systems) · Bernhard Schölkopf (MPI for Intelligent Systems/Amazon) · Dominik Janzing (Amazon)

Wednesday, 12/11 | 10:45-12:45pm | East Exhibition Hall B&C

On Single Source Robustness in Deep Fusion Models | #93
Taewan Kim (Amazon) · Joydeep Ghosh (UT Austin)

Perceiving the Arrow of Time in Autoregressive Motion | #155
Kristof Meding (University Tübingen) · Dominik Janzing (Amazon) · Bernhard Schölkopf (MPI for Intelligent Systems/Amazon) · Felix A. Wichmann (University of Tübingen)

Wednesday, 12/11 | 5:00-7:00pm | East Exhibition Hall B&C

Compositional De-Attention Networks | #127
Yi Tay (Nanyang Technological University) · Anh Tuan Luu (MIT) · Aston Zhang (Amazon) · Shuohang Wang (Singapore Management University) · Siu Cheung Hui (Nanyang Technological University)

Low-Rank Bandit Methods for High-Dimensional Dynamic Pricing | #3
Jonas Mueller (Amazon) · Vasilis Syrgkanis (Microsoft Research) · Matt Taddy (Amazon)

MaxGap Bandit: Adaptive Algorithms for Approximate Ranking | #4
Sumeet Katariya (Amazon/University of Wisconsin-Madison) · Ardhendu Tripathy (UW Madison) · Robert Nowak (UW Madison)

Primal-Dual Block Generalized Frank-Wolfe | #165
Qi Lei (UT Austin) · Jiacheng Zhuo (UT Austin) · Constantine Caramanis (UT Austin) · Inderjit S Dhillon (Amazon/UT Austin) · Alexandros Dimakis (UT Austin)

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling | #208
Tengyang Xie (University of Illinois at Urbana-Champaign) · Yifei Ma (Amazon) · Yu-Xiang Wang (UC Santa Barbara)

Thursday, 12/12 | 10:45-12:45pm | East Exhibition Hall B&C

AutoAssist: A Framework to Accelerate Training of Deep Neural Networks | #155
Jiong Zhang (UT Austin) · Hsiang-Fu Yu (Amazon) · Inderjit S Dhillon (UT Austin/Amazon)

Exponentially Convergent Stochastic k-PCA without Variance Reduction | #200 (oral, 10:05-10:20 W Ballroom C)
Cheng Tang (Amazon)

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift | #54
Stephan Rabanser (Technical University of Munich/Amazon) · Stephan Günnemann (Technical University of Munich) · Zachary Lipton (Carnegie Mellon University/Amazon)

High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes | #107
David Salinas (Naverlabs) · Michael Bohlke-Schneider (Amazon) · Laurent Callot (Amazon) · Jan Gasthaus (Amazon) · Roberto Medico (Ghent University)

Learning Search Spaces for Bayesian Optimization: Another View of Hyperparameter Transfer Learning | #30
Valerio Perrone (Amazon) · Huibin Shen (Amazon) · Matthias Seeger (Amazon) · Cedric Archambeau (Amazon) · Rodolphe Jenatton (Amazon)

Mo’States Mo’Problems: Emergency Stop Mechanisms from Observation | #227
Samuel Ainsworth (University of Washington) · Matt Barnes (University of Washington) · Siddhartha Srinivasa (University of Washington/Amazon)

Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting | #113
Rajat Sen (Amazon) · Hsiang-Fu Yu (Amazon) · Inderjit S Dhillon (UT Austin/Amazon)

Thursday, 12/12 | 5:00-7:00pm | East Exhibition Hall B&C

Dynamic Local Regret for Non-Convex Online Forecasting | #20
Sergul Aydore (Stevens Institute of Technology) · Tianhao Zhu (Stevens Institute of Technology) · Dean Foster (Amazon)

Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space | #47
Suo Yang (UT Austin), Yanyao Shen (UT Austin), Sujay Sanghavi (UT Austin/Amazon)

Inverting Deep Generative Models, One Layer at a Time |#48
Qi Lei (University of Texas at Austin) · Ajil Jalal (UT Austin) · Inderjit S Dhillon (UT Austin/Amazon) · Alexandros Dimakis (UT Austin)

Provable Non-linear Inductive Matrix Completion| #215
Kai Zhong (Amazon) · Zhao Song (UT Austin) · Prateek Jain (Microsoft Research) · Inderjit S Dhillon (UT Austin/Amazon)

Amazon researchers on NeurIPS committees and boards

  • Bernhard Schölkopf – Advisory Board
  • Michael I. Jordan – Advisory Board
  • Thorsten Joachims – senior area chair
  • Anshumali Shrivastava – area chair
  • Cedric Archambeau – area chair
  • Peter Gehler – area chair
  • Sujay Sanghavi – committee member

Workshops

Learning with Rich Experience: Integration of Learning Paradigms

Paper: "Meta-Q-Learning" | Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alexander J. Smola

Human-Centric Machine Learning

Paper: "Learning Fair and Transferable Representations" | Luco Oneto, Michele Donini, Andreas Maurer, Massimiliano Pontil

Bayesian Deep Learning

Paper: "Online Bayesian Learning for E-Commerce Query Reformulation" | Gaurush Hiranandani, Sumeet Katariya, Nikhil Rao, Karthik Subbian

Meta-Learning

Paper: "Constrained Bayesian Optimization with Max-Value Entropy Search" | Valerio Perrone, Iaroslav Shcherbatyi, Rodolphe Jenatton, Cedric Archambeau, Matthias Seeger

Paper: "A Quantile-Based Approach to Hyperparameter Transfer Learning" | David Salinas, Huibin Shen, Valerio Perrone

Paper: "A Baseline for Few-Shot Image Classification" | Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, Stefano Soatto

Conversational AI

Organizer: Dilek Hakkani-Tür

Paper: "The Eighth Dialog System Technology Challenge" | Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, Minlie Huang, Luis Lastras, Jonathan K. Kummerfeld, Walter S. Lasecki, Chiori Hori, Anoop Cherian, Tim K. Marks, Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta

Paper: “Just Ask: An Interactive Learning Framework for Vision and Language Navigation” | Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, Dilek Hakkani-Tur

Paper: “MA-DST: Multi-Attention-Based Scalable Dialog State Tracking” | Adarsh Kumar, Peter Ku, Anuj Kumar Goyal, Angeliki Metallinou, Dilek Hakkani-Tür

Paper: “Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery” | Maryam Fazel-Zarandi, Longshaokan Wang, Aditya Tiwari, Spyros Matsoukas

Paper: “Towards Personalized Dialog Policies for Conversational Skill Discovery”| Maryam Fazel-Zarandi, Sampat Biswas, Ryan Summers, Ahmed Elmalt, Andy McCraw, Michael McPhillips, John Peach

Paper: “Conversation Quality Evaluation via User Satisfaction Estimation” | Praveen Kumar Bodigutla, Spyros Matsoukas, Lazaros Polymenakos

Paper: “Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering” | Li Zhou, Kevin Small

Science Meets Engineering of Deep Learning

Paper: "X-BERT: eXtreme Multi-label Text Classification using Bidirectional Encoder from Transformers" Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, Inderjit S. Dhillon

Machine Learning with Guarantees

Organizers: Ben London, Thorsten Joachims
Program Committee: Kevin Small, Shiva Kasiviswanathan, Ted Sandler

MLSys: Workshop on Systems for ML

Paper: "Block-Distributed Gradient Boosted Trees" | Theodore Vasiloudis, Hyunsu Cho, Henrik Boström

Women in Machine Learning

Gold sponsor: Amazon

About the Author
Larry Hardesty is a science writer at Amazon. Previously, he was managing editor of the Boston Book Review, a senior editor at MIT Technology Review, and the computer science writer at the MIT News Office.

Work with us

See More Jobs
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services LLCTitle: Applied Scientist IIWorksite: Seattle, WAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Do you want to join Alexa AI -- the science team behind Amazon’s intelligence voice assistance system? Do you want to utilize cutting-edge deep-learning and machine learning algorithms to delight millions of Alexa users around the world?If your answers to these questions are “yes”, then come join us at the Alexa Artificial Intelligence team, which is in charge of improving Alexa user satisfaction through real-time metrics monitoring and continuous closed-loop learning. The team owns the modules that reduce user perceived defects and frictions through utterance reformulation, contextual and personalized hypothesis ranking.With the Alexa Artificial Intelligence team, you will be working alongside a team of experienced machine/deep learning scientists and engineers to create data driven machine learning models and solutions on tasks such as sequence-to-sequence query reformulation, graph feature embedding, personalized ranking, etc..You will be expected to:· Analyze, understand, and model user-behavior and the user-experience based on large scale data, to detect key factors causing satisfaction and dissatisfaction (SAT/DSAT).· Build and measure novel online & offline metrics for personal digital assistants and user scenarios, on diverse devices and endpoints· Create and innovate deep learning and/or machine learning based algorithms for utterance reformulation and contextual hypothesis ranking to reduce user dissatisfaction in various scenarios;· Perform model/data analysis and monitor user-experienced based metrics through online A/B testing;· Research and implement novel machine learning and deep learning algorithms and models.
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon Web Services, Inc.Job Title: Data Scientist IILocation: Seattle, WAPosition Responsibilities:Manage metrics reporting, perform data mining and big data analysis, to provide strategic advice on business and improve processes and systems. Collect business use cases, research and evaluate opportunities, and leverage data to support business functions through advanced mathematical modeling. Independently own and solve business problems. Transform business problems into mathematical models, provide data-driven solutions, and write clear and factually correct documents explaining technical concept to non-technical audience. Gather and use complex data sets across domains and proactively gather data when not readily available. Use specialized modelling software and scripting languages to scale methods. Partner with business, science, and engineering to establish scalable, efficient, automated processes for large scale data analyses, model development, model validation, and model implementation.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Workforce Staffing (WFS) is responsible for filling all of our First Mile, Middle Mile, and Last Mile Operations with hourly labor. In 2019, WFS hired hundred of thousands of hourly associates across NA and EU and will receive over millions of job applications for employment. This role is part of the Workforce Intelligence Team, tasked acquiring, modeling and visualizing all the data required to report out on performance metrics such as fill rates and funnel statistics, and forecasting hiring volumes to predict hiring risks and to support internal capacity planning.As part of the Workforce Intelligence team, the Ops Research Scientist will work on forecasting and optimization projects to improve funnel efficiency. The Ops Research Scientist will partner with capacity planning leaders, product/program managers, data engineers and other research scientists to build tools with clear business impact in an exciting and fast-paced start-up environment. The OR Scientist will be responsible for developing new predictive and optimization models as well as algorithms for key applications. The OR Scientist will be expected to be a thought leader as we chart new courses with our labor planning models. Successful candidates will have a deep knowledge of computational optimization methods and mathematical modeling, background in statistical and machine learning methods, the ability to map models into production-worthy code, the communication skills necessary to explain complex technical approaches to a variety of stakeholders and customers, and the excitement to take iterative approaches to tackle big, long term problems.Key Responsibilities:· Guide the technical approach for the design and implementation of successful models and algorithms in support of teams across Amazon.· Work in expert cross-functional teams delivering on demanding projects.· Functionally decompose complex problems into simple, straight-forward solutions.· Share knowledge in state-of-the-art statistics and machine learning research or frontier applied mathematical modeling and computation applicable to our problem space.
US, WA, Seattle
Within Amazon’s Corporate Financial Planning & Analysis team (FP&A), we enjoy a unique vantage point into everything happening within Amazon. As part of that, this role would be part of a team that is responsible for Company’s enterprise-wide financial planning & analytics environment.- Are you excited about working directly to empower users?- Love to get your hands dirty and solve challenging technical issues?We are looking for a customer obsessed Data Scientist who can apply the latest research, state of the art algorithms and machine learning to build highly scalable systems in the financial planning and analytics domain.The successful candidate will have strong data mining and modeling skills and is comfortable facilitating and working from concept through to execution. This role will also build tools and support structures needed to analyze data and present findings to business partners to drive improvements.The data flowing through our platform directly contributes to decision-making by our CFO and all levels of finance leadership. If you’re passionate about building tools that enhance productivity, improve financial accuracy, reduce waste, and improve work-life harmony for a large and rapidly growing finance user base, come join us! If you are passionate about solving complex problems, in a challenging environment, we would love to talk with you.Responsibilities of this position include:A qualified candidate must have demonstrated ability to manage modeling projects, identify requirements and build methodology and tools that are statistically grounded but also explainable operationally, apply technical skills allowing the models to adapt to changing attributes. In addition to the modeling and technical skills, possess strong written and verbal communication skills, strong focus on internal customers, and high intellectual curiosity with ability to learn new concepts/frameworks, algorithms and technology rapidly as changes arise.This is a tremendous opportunity to develop creative, new and innovative ways of interacting with and interpreting business key performance indicators.We're looking for innovators who want to create reliable and scalable solutions to help our leaders create the appropriate strategies.Additional responsibilities may include:· Research machine learning algorithms and implement by tailoring to particular business needs and tested on large datasets.· Predict future customer behavior and business conditions (Machine Learning, Predictive Modeling) and manipulating/mining data from database tables (Redshift, Oracle, Data Warehouse)· Create automated metrics· Providing analytical network support to improve quality and standard work results· Root cause research to identify process breakdowns within departments and providing data through use of various skill sets to find solutions to breakdown· Foster culture of continuous improvement for Customer Experience through all aspects of Data;(BI, Data Analytics, and Data Science)· Participate in the full development life cycle, end-to-end with cross functional teams, from design, implementation and testing, to documentation, delivery, support, and maintenance.As a member of our team you will be responsible for modelling complex problems, discovering insights and identifying opportunities through the use of statistical, machine learning, algorithmic, data mining and visualization techniques. You will need to collaborate effectively with internal stakeholders and cross-functional teams to solve problems, and create operational efficiencies. You should be able to apply a breadth of tools, data sources and analytical techniques to answer a wide range of high-impact business questions and present the insights in concise and effective manner. Additionally, you will need to be entrepreneurial, able to deal with high ambiguity and should be an effective communicator capable of independently driving issues to resolution and communicating insights to technical and non-technical audiences.
US, WA, Seattle
Voice-driven AI experiences are finally becoming a reality and Amazon’s Alexa voice cloud service and Echo devices are at the forefront of this latest technology wave. We deliver world-class products on aggressive schedules that are used every day, by people you know, in and about their homes. At the same time, we obsess about customer trust and ensure that we build products in a manner that maintains our high bar for customer privacy. We are looking for a passionate and talented data science leader to develop the strategies for understanding where accuracy gaps exist today, drive new scientific techniques to address them short- and long-term plan and establish the mechanisms for a successful execution.This is a unique opportunity to play a key role in an exciting, fast growing business. You come with a start-up mentality and skills that span research science, software engineering, product and strategic vision. You will work closely with talented engineers and lead ML scientists to put the algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally.You are the ideal candidate if you are clearly passionate about delivering experiences that delight customers and creating solutions that are robust. You should thrive in ambiguous environments that require to find solutions to problems that have not been solved before. You enjoy and succeed in fast paced environments where learning new concepts quickly is a must. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience building large-scale distributed systems to creating reliable, scalable, and high performance products. You provide technical and scientific guidance to your team members. Your strong communication skills enable you to work effectively with both business and technical partnersYou will be joining a select group of people making history producing one of the most highly rated products in Amazon's history.
US, WA, Seattle
Work at the intersection of data science and economics.The DAC AdsEcon Team is looking for a Data Scientist II to help and be part of a team to put cutting edge economic and data science advertising research into production. We are looking for a unique individual who is interested in bigger picture strategic thinking but with the passion for big data.Advertising is used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses econometrics, machine learning, and data science to help advertisers choose the right advertising product to meet their marketing goals. We also generate insights to guide Amazon Advertising strategy, providing direct support to the high level leaders.If you have a background in economics, computer science, statistics, or mathematics and have a passion for solving large, and impactful problems, this is the job for you. Key responsibilities of Data Scientist include the following:· Partnering with economists and senior team members to drive science improvements and implement technical solutions at the cutting edge of machine learning and econometrics· Helping build data systems that leverage diverse data sources to understand how different advertiser’s decisions impact their performance across multiple advertising products.· Build interpretable statistical models and analyze experiment results to answer questions that will drive high impact decisions across Amazon.About Amazon's Advertising business:Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.
US, WA, Seattle
About Prime Video: Prime Video is changing the way people watch movies and TV shows, offering more than 150,000 new release movies, next-day television shows and classic favorites available to rent or purchase on-demand, and more than 38,000 titles available to customers with an Amazon Prime membership. We believe so deeply in the mission of our video offering that we've launched our own Amazon Studios to create Original and Exclusive content. With an Amazon Prime membership, customers can have unlimited access to thousands of titles for no additional charge, including popular and award-winning Prime Originals like Jack Ryan, Fleabag and The Marvelous Mrs. Maisel.About the team: The vision for the engagement automation team is to inspire our customers to engage with all that Prime Video brand has to offer. To achieve our vision, we create product and technology solutions that drive incremental activation and engagement of PV customers worldwide. We obsess over finding effective ways to reach active and inactive customers with relevant and timely content that drives traffic to the PV experience. Using smart rules and machine learning we generate relevant, timely, and personalized engagement opportunities via a broad portfolio of both in-app and out-of-app experiences, on a fully automated basis.About the role: We seek an experienced Applied Scientist to be join Engagement Automation. Join us in defining and designing a fully automated E2E engagement system powered by science to increase customer engagement, activation and global adaption of prime video. You will have the opportunity to apply latest neural network, deep learning, and transfer learning models to define the target audience for Prime Video Originals and shows. You will also have the ability to apply causal modeling to identify customers with the most incremental impact from marketing activity as well as the defining characteristics of the most effective marketing campaigns. Additionally there is opportunity to apply Reinforcement Learning techniques to define the optimal marketing strategy (frequency, recency, touch point, channel, creative, copy) for each customer.You should expect to exercise both your coding skills and creative research thinking as you map real world processes to ML enabled systems. You will spend your time as a hands-on machine learning practitioner and a research leader. You will play a key role on the team, building and guiding machine learning models from the ground up. If you’re looking for an opportunity to make a big impact in a global business with a startup culture, we’re looking for you.
US, VA, Arlington
Amazon’s Talent Assessment team designs and implements hiring systems for one of the world’s fastest growing companies. We work in a data-focused, global environment solving complex problems with deep thought, large-sample research, and advanced quantitative methods to deliver practical solutions that make hiring more fair, accurate, and efficient.We're looking for an experienced assessment and personnel selection scientist who is equal parts researcher, consultant, and thought leader, with strong expertise in psychometrics, research methodology, and data analysis. In this role, you will collaborate with cross-functional teams of psychologists, ML scientists, UX researchers, engineers, and product managers, to direct the research, development, and implementation of new assessment methods to measure exactly what it requires to be an engaged and successful employee at Amazon.What you’ll do:· Lead the development and research of new content and approaches to assessment (e.g., high fidelity simulation, non-cognitive computer adaptive testing, interactive item types)· Design and execute large-scale, highly-visible global assessment validation and optimization projects· Develop assessment content, including personality, cognitive ability, and simulations· Perform complex statistical/quantitative analyses with large datasets· Apply the scientific method to answer novel research questions· Influence executive project sponsors and stakeholders across the company· Drive effective teamwork, communication, collaboration and commitment across cross-functional groups with competing priorities
US, WA, Seattle
The Economic Technology team (ET) is looking for a Senior Applied Scientist to join our team in building Reinforcement Learning solutions at scale. The ET applies Machine Learning, Reinforcement Learning, Causal Inference, and Econometrics/Economics to derive actionable insights about the complex economy of Amazon’s retail business. We also develop Statistical Models and Algorithms to drive strategic business decisions and improve operations. We are an interdisciplinary team of Economists, Engineers, and Scientists incubating and building day one solutions using cutting-edge technology, to solve some of the toughest business problems at Amazon.You will work with business leaders, scientists, and economists to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed services. You will partner with scientists, economists, and engineers to help invent and implement scalable ML, RL, and econometric models while building tools to help our customers gain and apply insights. This is a unique, high visibility opportunity for someone who wants to have business impact, dive deep into large-scale economic problems, enable measurable actions on the Consumer economy, and work closely with scientists and economists. We are particularly interested in candidates with experience building predictive models and working with distributed systems.As a Senior Applied Scientist, you bring business and industry context to science and technology decisions. You set the standard for scientific excellence and make decisions that affect the way we build and integrate algorithms. Your solutions are exemplary in terms of algorithm design, clarity, model structure, efficiency, and extensibility. You tackle intrinsically hard problems, acquiring expertise as needed. You decompose complex problems into straightforward solutions.
US, WA, Seattle
Do you want to join Alexa AI -- the science team behind Amazon’s intelligence voice assistance system? Do you want to utilize cutting-edge deep-learning and machine learning algorithms to delight millions of Alexa users around the world?If your answers to these questions are “yes”, then come join the Alexa Artificial Intelligence team. We are responsible for the deep learning technology that is central to the automated ranking and arbitration to optimize for end-to-end customer satisfaction.As an Applied Science Manager you will lead the science efforts to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. You will also:· Build a strong and coherent team with particular focus on automated ranking and arbitration, sciences, and innovation.· Serve as a technical lead on demanding and cross-team projects, and effectively collaborating with multiple cross-organizational teams· Apply technical influence on partner teams, increasing their productivity by sharing your deep knowledge.
US, WA, Seattle
Are you interested in delighting customers and are passionate about promoting sustainability? Then Amazon’s Packaging Team is the place for you. We are the Customer Packaging EXperience (CPEX) team and we optimize Amazon’s packaging solutions. To do this across billions of shipments, we are looking for someone with statistics and machine learning skills to design, test, implement and maintain packaging decision mechanisms across Amazon.You'll be responsible for the design, implementation, operation, and support of large-scale, performance-critical data science and machine learning systems which help to choose the optimal packaging types for our products. These technologies utilize every quadrant of data science including statistics, NLP, and CV to make the right decision on millions of products. You will work with scientists, software developers, business intelligence engineers, and product managers on our team as well as partner with packaging automation, concessions, and sustainability teams.
US, WA, Seattle
Have you ever ordered a product on Amazon and wondered how that box with a smile arrived at your doorstep so fast? Wondered where it came from and how much it cost Amazon? If so, the Amazon Global Supply Chain Optimization organization is for you.Watch this video to learn more about our organization, SCOT: http://bit.ly/amazon-scotWe are the most customer-centric company on Earth. We need exceptionally talented, bright, and driven people to continue to raise the bar on customer experience.Our objective is to build an experience where you can find and buy anything online and have it delivered fast! We continue to innovate with delivery speed initiatives so that Amazon will continue to own ‘fast’ in the minds of our customers. We are looking for a dynamic, organized self-starter to join as a Research Scientist, who will create state of the art models on Speed initiatives and programs and develop the measurement criteria for success.The AIM (Automated Inventory Management) team in the Supply Chain Optimization Technologies (SCOT) organization is dedicated to answering key strategic questions using quantitative and statistical methods. We develop cutting edge data pipelines, build accurate predictive models, and deploy automated software solutions to provide insights to business leaders at the most senior levels throughout the company. We are looking for a talented, driven, and analytical researcher to help us answer these strategic questions.The AIM Science team leads the way in developing innovative models, algorithms and strategies that will help us gain insights into how our business will grow and what will the drivers of such growth. These predictive models and insights will be based along products and product categories, customer segments, regions and locations, etc.This Research Scientist role will explore and develop innovative quantitative approaches and models, generate features, test hypotheses, design experiments, build predictive models, and work with very large complex data sets in order to explore relationship between business outcomes and key drivers, and then predict the trend on those business outcomes. These predictions and insights will provide a foundation of the highest level of visibility and importance for Amazon's financial and operational planning. The successful candidate will be a problem solver who enjoys diving into data, is excited by difficult modeling challenges, and possesses strong communication skills to effectively interface between technical and business teams, working together with Software Engineers, Product Managers, Business Analysts and other Scientists.Key Responsibilities:· Research, develop and build predictive models for Amazon business metrics with the goal of higher customer satisfactions. Analyze and research features and engineer features that help support predictive models to connect the dots among different functions such as inventory, speed of delivery, and best selections.. Provide insights by analyzing historical data· Constructively critique peer research and mentor junior scientists and engineers.· Create experiments and prototype implementations of new learning algorithms and prediction techniques.· Collaborate with engineering teams to design and implement software solutions for science problems.· Contribute to progress of the Amazon and broader research communities by producing publications.Amazon is an Equal Opportunity Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation.
US, WA, Virtual Location - Washington
Amazon has built a reputation for excellence with recent examples of being named the #1 most trusted company for customers. The Selling Partner Abuse team's mission is to protect the trust of our stores for customers and selling partners. We see ourselves as the steward of customer trust and Amazon brand.We are seeking a Machine Learning Science Manager who will lead a team of Applied Scientists, Research Scientists and Data scientists to research and develop innovative machine learning and science solutions to prevent the bad activities in the Amazon stores.Your responsibilities will include:· Build and develop the core science team for Selling Partner Abuse· Create new projects to drive significant business impact and research advancement· Create project milestone and manage multiple projects end-to-end while quickly adapt to changing priorities and generate innovative solutions in an extremely fast-paced environment· Coach the team and continuously raise the bar on highest standard· Build a strong partnership across different business, engineering and science stakeholders· Manage different Machine Learning projects that cover different science areas such as Ensemble Tree learning models, Clustering, Anomaly Detection, Graph models, NLP models, Semi-supervised Learning models and Reinforcement Learning
AU, VIC, Melbourne
Amazon delights millions of customers around the world. Meet the behind the scenes team that enables our Human Resource and Operations Leaders to make informed decisions. The Amazon PeopleInsight team builds reporting and analytics tools for our teams that fulfill customer promise every day. Whether it is Fulfillment Center team that delivers your Prime order in two days, our Amazon Locker team that lets you pick up your package anytime that is convenient for you, our Prime Now team getting you lunch in under an hour, or one of many more, the PeopleInsight group is there providing people metrics along the employee lifecycle for our global operations businesses. The PeopleInsight team is a collaborative group of Business Analysts, Business Intelligence Engineers, Data Engineers, Data Scientists, Product Managers, and Program Managers dedicated to empowering leaders and enabling action through data and science. We deliver workforce, associate experience, and leadership insights so Amazon leaders can focus their efforts in ways that will engage, retain and grow their associates.We are now recruiting for an exceptional Data Scientist, Worldwide OperationsThe ideal candidate will be:· A Well-Rounded Athlete –Like a true athlete, you understand that we succeed or fail as a team. You are always ready to step up beyond your core responsibilities and go the extra mile for the project and your team. You nimbly overcome barriers to deliver the best products more quickly than expected.· A Perpetual Student – You seek knowledge and insight. You challenge yourself to turn moments into master’s classes. Whether closing a gap, developing a new skill, or staying ahead of your industry, you revel in the joy of learning and growing.· A Skilled Communicator – You excel when interacting with business and technical partners whether you are chatting, sending a written message, or conducting a presentation.· A Trusted Advisor – You work closely with stakeholders to define key business needs and deliver on commitments. You enable effective decision making by retrieving and aggregating data from multiple sources and compiling it into a digestible and actionable format.· An Inventor at Heart – You innovate on behalf of your customer by proactively implementing improvements, enhancements, and customizations. Your customers marvel at your creative solutions to challenges they had not yet identified.· A Fearless Explorer – You are drawn to take on the hardest problems, navigate ambiguity, and battle skepticism. You never settle, even in the face of overwhelming obstacles.Roles and ResponsibilitiesSuccess in this role will include influencing within your team and mentoring peers. The problems you will consider will be difficult to solve and often require a range of data science methodologies combined with subject matter expertise. You will need to be capable of gathering and using complex data set across domains. You will deliver artifacts on medium size projects, define the methodology, and own the analysis. Your findings will affect important business decisions. Solutions are testable and reproducible. You will create documents and share findings in line with scientific best practices for both technical and nontechnical audiences.Amazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation.
US, CA, Virtual Location - California
The Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, data scientists and MBAʼs. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement.Roughly 50% of research assistants from previous cohorts have converted to full time data science or economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.Amazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation
US, CA, Palo Alto
The Amazon Search team creates powerful, customer-focused search and advertising solutions and technologies. Whenever a customer visits an Amazon site worldwide and types in a query or browses through product categories, A9 Product Search services go to work. We design, develop, and deploy high performance, fault-tolerant distributed search systems used by millions of Amazon customers every day. Our Search team works to maximize the quality and trustworthiness of the search experience for visitors to Amazon websites worldwide.Our mission is to provide customers' trust and confidence in Amazon Search shopping experience. We identify problems that are customer trust busters at Amazon, deliver scalable and responsive solutions to these issues, and build experiences that gain customer trust using advanced machine learning methods. We carefully monitor the trustworthiness of the search results and dive deep when we see an unusual pattern. Most of the models used by our team is semi-supervised or unsupervised using small amount of labeled data.In this role you will leverage your strong statistical background to help build the next generation of our machine learning methods to discover untrustworthy search engagements, unsual patterns, and estimate a probability of risk for each item. This role requires a pragmatic technical leader comfortable with ambiguity, capable of summarizing complex data and models through clear visual and written explanations. The ideal candidate will have experience with machine learning models, graph algorithms, and information retrieval algorithms at scale. Additionally, we are seeking candidates with strong rigor in applied sciences and engineering, creativity, curiosity, and great judgment.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses.If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you.Major responsibilities· · Use machine learning and analytical techniques to create scalable solutions for business problems· · Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes· · Design, development, evaluate and deploy innovative and highly scalable models for predictive learning· · Research and implement novel machine learning and statistical approaches· · Work closely with software engineering teams to drive real-time model implementations and new feature creations· · Work closely with business owners and operations staff to optimize various business operations· · Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation· · Mentor other scientists and engineers in the use of ML techniques
US, MA, Cambridge
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services LLCJob Title: Applied Scientist IILocation: Cambridge, MAPosition Responsibilities:Designing, developing, evaluating, deploying, and updating data-driven models and analytical solutions for speech and language understanding applications. Developing practical solutions to complex problems. Translating business and functional requirements into concrete deliverables and building quick prototypes or proofs of concept with team members. Solving technical challenges related to speech and language understanding technologies. Mentoring junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
IN, KA, Bangalore
Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.The Moderation and Relevance System (MARS) team, based in Bangalore, is responsible for ensuring that ads are relevant and is of good quality, leading to higher conversion for the sellers and providing a great experience for the customers. We deal with one of the world’s largest product catalog, handle billions of requests a day with plans to grow it by order of magnitude and use automated systems to validate tens of millions of offers submitted by thousands of merchants in multiple countries and languages.In this role, you will build and develop ML models to address content intelligence problems, build advanced algorithms in detecting and generating content. These models will rely on a variety of visual and textual features requiring expertise in both domains. These models need to scale to multiple languages and countries. You will collaborate with engineers and other scientists to build, train and deploy these models. You will propose hypotheses, validate these offline and run A/B tests to validate them online. As part of these activities, you will develop production level code that enables moderation of millions of ads submitted each day.