Vancouver, Canada

3 important themes from Amazon's 2019 NeurIPS papers

Time series forecasting, bandit problems, and optimization are integral to Amazon's efforts to deliver better value for its customers.

Last year, the first 2,000-2,500 publicly released tickets to the Conference on Neural Information Processing Systems, or NeurIPS, sold out in 12 minutes.

This year, the conference organizers moved to a lottery system, allowing aspiring attendees to register in advance and randomly selecting invitees from the pool of registrants. But they also bumped the number of public-release tickets up from around 2,000 to 3,500, testifying to the conference’s continued popularity.

At NeurIPS this year, there are 26 papers with Amazon coauthors. They cover a wide range of topics, but surveying their titles, Alex Smola, a vice president and distinguished scientist in the Amazon Web Services organization, discerns three prominent themes, all tied to Amazon’s efforts to deliver better value for its customers.

Those three themes are time series forecasting (and causality), bandit problems, and optimization.

1. Time series forecasting

Time series forecasting involves measuring some quantity over time — such as the number of deliveries in a particular region in the past six months, or the number of cloud servers required to support a particular site over the past two years — and attempting to project that quantity into the future.

“That’s something that is very dear to Amazon’s heart,” Smola says. “For anything that Amazon does, it’s really beneficial to have a good estimate of what our customers will expect from us ahead of time. Only by being able to do that will we be able to satisfy customers’ demands, be it for products or services.”

A sequence of basis time series, forecast into the near future and summed together to approximate a new time series.
The paper “Think Globally, Act Locally” examines data sets with many correlated time series, such as the demand curves for millions of products sold online. The researchers describe a method for constructing a much smaller set of “basis time series”; the time series for any given product can be approximated by a weighted sum of the bases.
Courtesy of the researchers

The basic mathematical framework for time series forecasting is a century old, but the scale of modern forecasting problems calls for new analytic techniques, Smola says.

“Problems are nowadays highly multivariate,” Smola says. “If you look at the many millions of products that we offer, you want to be able to predict fairly well what will sell, where and to whom.

“You need to make reasonable assumptions on how this very large problem can be decomposed into smaller, more tractable pieces. You make structural approximations, and sometimes those structural approximations are what leads to very different algorithms.

“So you might, for instance, have a global model, and then you have local models that address the specific items or address the specific sales. If you look at ‘Think Globally, Act Locally’” — a NeurIPS paper whose first author is Rajat Sen, an applied scientist in the Amazon Search group — “it’s already in the title. Or look at ‘High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes’. In this case, you have a global structure, but it’s only in a small subspace where interesting things happen.”

Side-by-side images depict correlations between taxi traffic at different points in Manhattan at different times of day
The paper "High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes" describes a method for predicting correlations among many parallel time series. In one example, the researchers forecast correlations between the taxi traffic at different points in New York City at different times of day. Red lines indicate strong correlations; blue lines indicate strong negative correlations. Weekend midday traffic patterns (left) show negative correlations between locations near the Empire State Building, suggesting that taxis tend to prefer different routes depending on traffic conditions. Weekend evening traffic patterns show positive correlations between the vicinity of the Empire State Building and areas with high concentrations of hotels.
Courtesy of the researchers

An aspect of forecasting that has recently been drawing more attention, Smola says, is causality. Where traditional machine learning models merely infer statistical correlations between data points, “it is ultimately the causal relationship that matters,” Smola says.

“I think that causality is one of the most interesting conceptual developments affecting modern machine learning,” says Bernhard Schölkopf, like Smola a vice president and distinguished scientist in Amazon Web Services. “This is the main topic that I have been interested in for the last decade.”

Two of Schölkopf’s NeurIPS papers — “Perceiving the Arrow of Time in Autoregressive Motion” and “Selecting Causal Brain Features with a Single Conditional Independence Test per Feature” — address questions of causality, as does “Causal Regularization”, a paper by Dominik Janzing, a senior research scientist in Smola’s group.

“Normal machine learning builds on correlations of other statistical dependences,” Schölkopf explains. “This is fine as long as the source of the data doesn't change. For example, if in the training set of an image recognition system, all cows are standing on green pasture, then it is fine for an ML system to use the green as a useful feature in recognizing cows, as long as the test set looks the same. If in the test set, the cows are standing on the beach, then such a purely statistical system can fail.

“More generally: causal learning and inference attempts to understand how systems respond to interventions and other changes, and not just how to predict data that looks more or less the same as the training data.”

2. Bandit problems

The second major theme that Smola discerns in Amazon scientists’ NeurIPS papers is a concern with bandit problems, a phrase that shows up in the titles of Amazon papers such as “MaxGap Bandit: Adaptive Algorithms for Approximate Ranking” and “Low-Rank Bandit Methods for High-Dimensional Dynamic Pricing”. Bandit problems take their name from one-armed bandits, or slot machines.

“It used to be that those bandits were all mechanical, so there would be slight variations between them, and some would have maybe a slightly a higher return than others,” Smola explains. “I walk into a den of iniquity, and I want to find the one-armed bandit where I will lose the least money or maybe make some money. And the only feedback I have is that I pull arms, and I get money or lose money. These are very unreliable, noisy events.”

Bandit problems present what’s known as an explore-exploit trade-off. The gambler must simultaneously explore the environment — determine which machines pay out the most — and exploit the resulting knowledge — concentrate as much money as possible on the high-return machines. Early work on bandit problems concerned identifying the high-return machines with minimal outlays.

“That problem was solved about 20 years ago,” Smola says. “What hasn’t been solved — and this is where things get a lot more interesting — is once you start adding context. Imagine that I get to show you various results as you’re searching for your next ugly Christmas sweater. The unfortunate thing is that the creativity of sweater designers is larger than what you can fit on a page. Now the context is essentially, what time, where from, which user, all those things. We want to find and recommend the ugly Christmas sweater that works specifically for you. This is an example where context is immediately relevant.”

It’s really beneficial to have a good estimate of what our customers will expect from us ahead of time. Only by being able to do that will we be able to satisfy customers’ demands.
Alex Smola, VP and distinguished scientist, Amazon

In the bandit-problem framework, in other words, the high-payout machines change with every new interaction. But there may be external signals that indicate how they’re changing.

Distributed computing, which is inescapable for today’s large websites, changes the structure of the bandit problem, too.

“Say you go to a restaurant, and the cook wants to improve the menu,” Smola says. “You can try out lots of new menu items, and that’s a good way to improve the menu overall. But if you start offering a lot of undercooked dishes because you’re experimenting, then at some point your loyal customers will stay away.

“Now imagine you have 100 restaurants, and they all do the same thing at the same time. They can’t necessarily communicate at the per-second level; maybe every day or every week they chat with each other. Now this entire exploration problem becomes a little more challenging, because if two restaurants try out the same undercooked dish, you make the customer less happy than you could have.

“So how does this map back into Amazon land? Well, if you have many servers doing this recommendation, the explore-exploit trade-off might be too aggressive if every one of them works on their own.”

3. Optimization

Finally, Smola says, “There is a third category of results that has to do with making algorithms faster. If you look at ‘Primal-Dual Block Frank-Wolfe’, ‘Communication-Efficient Distributed SGD with Sketching’, ‘Qsparse-Local-SGD’ — those are the workhorses that run underneath all of this. Making them more efficient is obviously something that we care about, so we can respond to customer requests faster, train algorithms faster.”

Bird’s-eye view

NeurIPS is a huge conference, with more than 1,400 accepted papers that cover a bewildering variety of topics. Beyond the Amazon papers, Caltech professor and Amazon fellow Pietro Perona identifies three research areas as growing in popularity.

“One is understanding how deep networks work, so that we can better design architectures and optimization algorithms to train models,” Perona says. “Another is low-shot learning. Machines are still much less efficient than humans at learning, in that they need more training examples to achieve the same performance. And finally, AI and society — identifying opportunities for social good, sustainable development, and the like.”

NeurIPS is being held this year at the Vancouver Convention Center, and the main conference runs from Dec. 8 to Dec. 12. The Women in Machine Learning Workshop, for which Amazon is a gold-level sponsor, takes place on Dec. 9; the Third Conversational AI workshop, whose organizers include Alexa AI principal scientist Dilek Hakkani-Tür, will be held on Dec. 14.

Amazon's involvement at NeurIPS

Paper and presentation schedule

Tuesday, 12/10 | 10:45-12:45pm | East Exhibition Hall B&C

A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning | #192
Francisco Garcia (UMass Amherst/Amazon) · Philip Thomas (UMass Amherst)

Blocking Bandits | #17
Soumya Basu (UT Austin) · Rajat Sen (UT Austin/Amazon) · Sujay Sanghavi (UT Austin/Amazon) · Sanjay Shakkottai (UT Austin)

Causal Regularization | #180
Dominik Janzing (Amazon)

Communication-Efficient Distributed SGD with Sketching | #81
Nikita Ivkin (Amazon) · Daniel Rothchild (University of California, Berkeley) · Md Enayat Ullah (Johns Hopkins University) · Vladimir Braverman (Johns Hopkins University) · Ion Stoica (UC Berkeley) · Raman Arora (Johns Hopkins University)

Learning Distributions Generated by One-Layer ReLU Networks | #49
Shanshan Wu (UT Austin) ·Alexandros G. Dimakis (UT Austin) · Sujay Sanghavi (UT Austin/Amazon)

Tuesday, 12/10 | 5:30-7:30pm | East Exhibition Hall B&C

Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control | #195
Sai Qian Zhang (Harvard University) · Qi Zhang (Amazon) · Jieyu Lin (University of Toronto)

Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products | #37
Tharun Kumar Reddy Medini (Rice University) · Qixuan Huang (Rice University) · Yiqiu Wang (Massachusetts Institute of Technology) · Vijai Mohan (Amazon) · Anshumali Shrivastava (Rice University/Amazon)

Iterative Least Trimmed Squares for Mixed Linear Regression | #50
Yanyao Shen (UT Austin) · Sujay Sanghavi (UT Austin/Amazon)

Meta-Surrogate Benchmarking for Hyperparameter Optimization | #6
Aaron Klein (Amazon) · Zhenwen Dai (Spotify) · Frank Hutter (University of Freiburg) · Neil Lawrence (University of Cambridge) · Javier Gonzalez (Amazon)

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations | #32
Debraj Basu (Adobe) · Deepesh Data (UCLA) · Can Karakus (Amazon) · Suhas Diggavi (UCLA)

Selecting Causal Brain Features with a Single Conditional Independence Test per Feature | #139
Atalanti Mastakouri (Max Planck Institute for Intelligent Systems) · Bernhard Schölkopf (MPI for Intelligent Systems/Amazon) · Dominik Janzing (Amazon)

Wednesday, 12/11 | 10:45-12:45pm | East Exhibition Hall B&C

On Single Source Robustness in Deep Fusion Models | #93
Taewan Kim (Amazon) · Joydeep Ghosh (UT Austin)

Perceiving the Arrow of Time in Autoregressive Motion | #155
Kristof Meding (University Tübingen) · Dominik Janzing (Amazon) · Bernhard Schölkopf (MPI for Intelligent Systems/Amazon) · Felix A. Wichmann (University of Tübingen)

Wednesday, 12/11 | 5:00-7:00pm | East Exhibition Hall B&C

Compositional De-Attention Networks | #127
Yi Tay (Nanyang Technological University) · Anh Tuan Luu (MIT) · Aston Zhang (Amazon) · Shuohang Wang (Singapore Management University) · Siu Cheung Hui (Nanyang Technological University)

Low-Rank Bandit Methods for High-Dimensional Dynamic Pricing | #3
Jonas Mueller (Amazon) · Vasilis Syrgkanis (Microsoft Research) · Matt Taddy (Amazon)

MaxGap Bandit: Adaptive Algorithms for Approximate Ranking | #4
Sumeet Katariya (Amazon/University of Wisconsin-Madison) · Ardhendu Tripathy (UW Madison) · Robert Nowak (UW Madison)

Primal-Dual Block Generalized Frank-Wolfe | #165
Qi Lei (UT Austin) · Jiacheng Zhuo (UT Austin) · Constantine Caramanis (UT Austin) · Inderjit S Dhillon (Amazon/UT Austin) · Alexandros Dimakis (UT Austin)

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling | #208
Tengyang Xie (University of Illinois at Urbana-Champaign) · Yifei Ma (Amazon) · Yu-Xiang Wang (UC Santa Barbara)

Thursday, 12/12 | 10:45-12:45pm | East Exhibition Hall B&C

AutoAssist: A Framework to Accelerate Training of Deep Neural Networks | #155
Jiong Zhang (UT Austin) · Hsiang-Fu Yu (Amazon) · Inderjit S Dhillon (UT Austin/Amazon)

Exponentially Convergent Stochastic k-PCA without Variance Reduction | #200 (oral, 10:05-10:20 W Ballroom C)
Cheng Tang (Amazon)

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift | #54
Stephan Rabanser (Technical University of Munich/Amazon) · Stephan Günnemann (Technical University of Munich) · Zachary Lipton (Carnegie Mellon University/Amazon)

High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes | #107
David Salinas (Naverlabs) · Michael Bohlke-Schneider (Amazon) · Laurent Callot (Amazon) · Jan Gasthaus (Amazon) · Roberto Medico (Ghent University)

Learning Search Spaces for Bayesian Optimization: Another View of Hyperparameter Transfer Learning | #30
Valerio Perrone (Amazon) · Huibin Shen (Amazon) · Matthias Seeger (Amazon) · Cedric Archambeau (Amazon) · Rodolphe Jenatton (Amazon)

Mo’States Mo’Problems: Emergency Stop Mechanisms from Observation | #227
Samuel Ainsworth (University of Washington) · Matt Barnes (University of Washington) · Siddhartha Srinivasa (University of Washington/Amazon)

Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting | #113
Rajat Sen (Amazon) · Hsiang-Fu Yu (Amazon) · Inderjit S Dhillon (UT Austin/Amazon)

Thursday, 12/12 | 5:00-7:00pm | East Exhibition Hall B&C

Dynamic Local Regret for Non-Convex Online Forecasting | #20
Sergul Aydore (Stevens Institute of Technology) · Tianhao Zhu (Stevens Institute of Technology) · Dean Foster (Amazon)

Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space | #47
Suo Yang (UT Austin), Yanyao Shen (UT Austin), Sujay Sanghavi (UT Austin/Amazon)

Inverting Deep Generative Models, One Layer at a Time |#48
Qi Lei (University of Texas at Austin) · Ajil Jalal (UT Austin) · Inderjit S Dhillon (UT Austin/Amazon) · Alexandros Dimakis (UT Austin)

Provable Non-linear Inductive Matrix Completion| #215
Kai Zhong (Amazon) · Zhao Song (UT Austin) · Prateek Jain (Microsoft Research) · Inderjit S Dhillon (UT Austin/Amazon)

Amazon researchers on NeurIPS committees and boards

  • Bernhard Schölkopf – Advisory Board
  • Michael I. Jordan – Advisory Board
  • Thorsten Joachims – senior area chair
  • Anshumali Shrivastava – area chair
  • Cedric Archambeau – area chair
  • Peter Gehler – area chair
  • Sujay Sanghavi – committee member

Workshops

Learning with Rich Experience: Integration of Learning Paradigms

Paper: "Meta-Q-Learning" | Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alexander J. Smola

Human-Centric Machine Learning

Paper: "Learning Fair and Transferable Representations" | Luco Oneto, Michele Donini, Andreas Maurer, Massimiliano Pontil

Bayesian Deep Learning

Paper: "Online Bayesian Learning for E-Commerce Query Reformulation" | Gaurush Hiranandani, Sumeet Katariya, Nikhil Rao, Karthik Subbian

Meta-Learning

Paper: "Constrained Bayesian Optimization with Max-Value Entropy Search" | Valerio Perrone, Iaroslav Shcherbatyi, Rodolphe Jenatton, Cedric Archambeau, Matthias Seeger

Paper: "A Quantile-Based Approach to Hyperparameter Transfer Learning" | David Salinas, Huibin Shen, Valerio Perrone

Paper: "A Baseline for Few-Shot Image Classification" | Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, Stefano Soatto

Conversational AI

Organizer: Dilek Hakkani-Tür

Paper: "The Eighth Dialog System Technology Challenge" | Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, Minlie Huang, Luis Lastras, Jonathan K. Kummerfeld, Walter S. Lasecki, Chiori Hori, Anoop Cherian, Tim K. Marks, Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta

Paper: “Just Ask: An Interactive Learning Framework for Vision and Language Navigation” | Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, Dilek Hakkani-Tur

Paper: “MA-DST: Multi-Attention-Based Scalable Dialog State Tracking” | Adarsh Kumar, Peter Ku, Anuj Kumar Goyal, Angeliki Metallinou, Dilek Hakkani-Tür

Paper: “Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery” | Maryam Fazel-Zarandi, Longshaokan Wang, Aditya Tiwari, Spyros Matsoukas

Paper: “Towards Personalized Dialog Policies for Conversational Skill Discovery”| Maryam Fazel-Zarandi, Sampat Biswas, Ryan Summers, Ahmed Elmalt, Andy McCraw, Michael McPhillips, John Peach

Paper: “Conversation Quality Evaluation via User Satisfaction Estimation” | Praveen Kumar Bodigutla, Spyros Matsoukas, Lazaros Polymenakos

Paper: “Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering” | Li Zhou, Kevin Small

Science Meets Engineering of Deep Learning

Paper: "X-BERT: eXtreme Multi-label Text Classification using Bidirectional Encoder from Transformers" Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, Inderjit S. Dhillon

Machine Learning with Guarantees

Organizers: Ben London, Thorsten Joachims
Program Committee: Kevin Small, Shiva Kasiviswanathan, Ted Sandler

MLSys: Workshop on Systems for ML

Paper: "Block-Distributed Gradient Boosted Trees" | Theodore Vasiloudis, Hyunsu Cho, Henrik Boström

Women in Machine Learning

Gold sponsor: Amazon

About the Author
Larry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.

Recommended for you

Work with us

See More Jobs
US, WA, Seattle
We are growing our collaborative group of engineers and applied scientists by expanding into new areas. Come and join us as we invent new ways to delight Amazon customers.Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products.As Launch Day Search Quality you will be the single threaded owner who will hire and grow a new team to drive improvements and mechanisms for Global Search Quality in (to-be-launched) international locales. You will invent mechanisms to evaluate and launch improvements for locales with a vast variety of languages, traffic patterns, cultures and preferences. You and your team will invent novel techniques that improve state of the art systems for low resource settings. You will invent and deploy techniques that work across languages, locales and different shopping modes.Do no hesitate to reach out if you have some of the following: ability to apply state of the art in large scale Machine Learning (e.g. semi-weakly-un-supervised deep learning, natural language understanding, self-supervised learning), curiosity to learn through controlled experimentation and/or experience with low latency production systems.You and your team will be able to deliver measure your impact for customers of the most popular shopping destination on this planet.We are an inclusive employer and value diversity at Amazon. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.scijobs
US, WA, Seattle
We are growing our collaborative group of engineers and applied scientists by expanding into new areas. Come and join us as we invent new ways to delight Amazon customers.We are Amazon's Global Search Quality team. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products.As an. Applied Scientist you will push boundaries in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する).Do no hesitate to reach out if you have some of the following: ability to apply state of the art in large scale Machine Learning (e.g. semi-weakly-un-supervised deep learning, natural language understanding), curiosity to learn through controlled experimentation or experience with low latency production systems.Like everyone else on the team, you will be regularly rewarded by measurable impact on customers like ourselves.We are an inclusive employer and value diversity at Amazon. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.Apply now or ping Vaclav Petricek (https://www.linkedin.com/in/petricek) to learn more about the different ways you can have huge impact with us.scijobs
US, WA, Seattle
Work at the intersection of data science and economics.The DAC AdsEcon Team is looking for a Data Scientist III to help and be part of a team to put cutting edge economic and data science advertising research into production. We are looking for a unique individual who is interested in bigger picture strategic thinking but with the passion for big data.Advertising is used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses econometrics, machine learning, and data science to help advertisers choose the right advertising product to meet their marketing goals. We also generate insights to guide Amazon Advertising strategy, providing direct support to the high level leaders.If you have a background in economics, computer science, statistics, or mathematics and have a passion for solving large, and impactful problems, this is the job for you. Key responsibilities of Data Scientist include the following:· Partnering with economists and senior team members to drive science improvements and implement technical solutions at the cutting edge of machine learning and econometrics· Helping build data systems that leverage diverse data sources to understand how different advertiser’s decisions impact their performance across multiple advertising products.· Build interpretable statistical models and analyze experiment results to answer questions that will drive high impact decisions across Amazon.About Amazon's Advertising business:Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.
US, MA, Cambridge
Alexa is the groundbreaking voice service that powers Echo and other devices designed around your voice. Our team is creating the science and technology behind Alexa. We’re working hard, having fun, and making history. Come join our team! You will have an enormous opportunity to impact the customer experience, design, architecture, and implementation of a cutting edge product used every day by people you know.We’re looking for a passionate, talented, and inventive scientist to help build industry-leading conversational technologies that customers love. Our team's mission is the enable Alexa to understand sounds and vocalization beyond speech. As an Applied Scientist, you will work with talented peers to develop novel machine learning algorithms and modeling techniques to advance the state of the art in speech and audio processing. Your work will directly impact our customers in the form of novel products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding. You will mentor junior scientists, create and drive new initiatives.
US, CA, Cupertino
Are you passionate about applying formal verification, program analysis, constraint-solving, and theorem proving to real world problems? Do you want to create products that help customers? If so, then we have an exciting opportunity for you. In this role, you will interact with internal teams and external customers to understand their requirements. You will apply your knowledge to propose innovative solutions, create software prototypes, and productize prototypes into production systems using software development tools and methodologies. In addition, you will support and scale your solutions to meet the ever growing demand of customer use.Technical Responsibilities:- Interact with various teams to develop an understanding of their security and safety requirements.- Apply the acquired knowledge to build tools find problems, or show the absence of security/safety problems.- Implement these tools through the use of SAT, SMT, BDDs, and various concepts from programming languages, theorem proving, formal verification and constraint solving.- Perform analysis of the customer systems using tools developed in-house or externally provided- Create software prototypes to verify and validate the devised solutions methodologies; integrate the prototypes into production systems using standard software development tools and methodologies.Leadership Responsibilities:- Can present and defend company-wide technical decisions to the internal technical community and represent the company effectively at technical conferences.- Functional thought leader, sought after for key tech decisions. Can successfully sell ideas to an executive level decision maker.- Mentors and trains the research scientist community on complex technical issues.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us
US, CA, Cupertino
Are you passionate about applying formal verification, program analysis, constraint-solving, and theorem proving to real world problems? Do you want to create products that help customers? If so, then we have an exciting opportunity for you. In this role, you will interact with internal teams and external customers to understand their requirements. You will apply your knowledge to propose innovative solutions, create software prototypes, and productize prototypes into production systems using software development tools and methodologies. In addition, you will support and scale your solutions to meet the ever growing demand of customer use.Technical Responsibilities:- Interact with various teams to develop an understanding of their security and safety requirements.- Apply the acquired knowledge to build tools find problems, or show the absence of security/safety problems.- Implement these tools through the use of SAT, SMT, BDDs, and various concepts from programming languages, theorem proving, formal verification and constraint solving.- Perform analysis of the customer systems using tools developed in-house or externally provided- Create software prototypes to verify and validate the devised solutions methodologies; integrate the prototypes into production systems using standard software development tools and methodologies.Leadership Responsibilities:- Can present and defend company-wide technical decisions to the internal technical community and represent the company effectively at technical conferences.- Functional thought leader, sought after for key tech decisions. Can successfully sell ideas to an executive level decision maker.- Mentors and trains the research scientist community on complex technical issues.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us
DE, BE, Berlin
As a Senior Applied Scientist on this growing team, you will take on a key role in improving the NLP and ranking capabilities of the Amazon product search engine. Our ultimate goal is to help customers find the products they are searching for, and discover new products they would be interested in. We do so by developing NLP components that cover a wide range of languages, not only English and major languages of Europe, but also Turkish, Arabic, Japanese, and more. The team plays a central role in search query understanding, product indexing, and representations/embeddings of queries and products, all of which aid in improving the ranking and relevance of search results.This is a rewarding role where you will be able to draw a clear connection between your work and how it improves the experience of millions of Amazon customers across the globe every day. You will propose and explore publication-worthy innovation in NLP and IR to build ML models trained on terabytes of product and traffic data, which are evaluated using both offline metrics as well as online metrics from A/B testing. You will then integrate these models into the production search engine that serves customers, closing the loop through data, modeling, application, and customer feedback. The chosen approaches for model architecture will balance business-defined performance metrics with the needs of millisecond response times.Your responsibilities include:· Analyze the data and metrics resulting from traffic into Amazon's product search engine· Design, build, and deploy effective and innovative ML solutions to improve various components of the search stack, such as indexing, ranking, and query understanding· Evaluate the proposed solutions via offline benchmark tests as well as online A/B tests in production· Publish and present your work at internal and external scientific venues in the fields of ML/NLP/IRYour benefits include:· Working on a high-impact, high-visibility product, with your work improving the experience of millions of customers· The opportunity to use (and innovate) state-of-the-art ML methods to solve real-world problems· Being part of a growing team where you can influence the team's mission, direction, and how we achieve our goals· Excellent opportunities, and ample support, for career growth, development, and mentorship· Competitive compensation, including relocation support (for both domestic and international candidates)
US, WA, Seattle
Do you have the passion to drive performance of a business through creative analytical insight? Are you excited about Human Resources and bringing science to the art of managing and developing an extremely talented workforce? We are looking for a passionate, insightful, results-oriented Analyst to join a team chartered with building Human Resources Analytics for all of Devices & Services. From statistical analysis, predictive analytics, product building for reporting designed to scale, and requirements gathering, we do it. All with an aim to more effectively attract and develop our most valuable of all resources – our people.The Data Science, Workforce Insights role is ideal for someone with the analytical acumen to discover and communicate valuable staffing and workforce insights and recommendations.In this role, you will closely partner with the Devices & Services HR and Talent Acquisition leadership team. You will also work closely with cross-functional teams and leaders to understand business goals and priorities, in order to effectively and accurately strategize, plan and execute initiatives as it relates to workforce plans. Excellent interpersonal & communication skills, the ability to influence at an executive level, and the ability to tell customized end-to-end headcount progress-to-goal and movement story will be critical for success.This role performs advanced business analysis using various techniques and modeling (e.g. statistical, explanatory and predictive modeling and SQL data mining).The Team: How often have you had an opportunity to be a member of a team innovating and solving some of the world's toughest problems? You can expect all the challenges and benefits of a growing business: lots of room for improvement and innovation, a close-knit team that cares about one another as humans, and a fast-paced environment. These will require a willingness to dive into the details, solve new problems as they arise, leverage high judgment and gut instinct, look around corners, and always obsess over customers. Everyone on the team wears multiple hats, and ownership our end-products is key.Responsibilities:· Collaborate with HR, business leaders, and peer analysts/researchers on hypothesis-driven research projects to inform business actions· Design, develop and evaluate Talent Acquisition (TA) business reporting and insights· Establish scalable, efficient, automated, and easily repeated processes for large scale data analyses for the broader Amazon Devices org· Provide ad-hoc or special project statistical analysis to TA, HR and business leaders that enable them to achieve their goals, drive decision making and create new strategies where necessary· Gather, manage, and evaluate large datasets from multiple sources· Build and maintain strong partnerships across TA, HR and the business
US, WA, Seattle
Ever wonder how you can apply your analytics skills to drive customer experience improvements for a diverse, innovative, and fun company like Amazon? Amazon’s Customer Insights and Research team is looking for a Data Scientist to help do exactly that and much more! If you have experience in surveys, experimental design, statistical analysis, and NLP-based machine learning algorithms and are interested in going deeper into data, this is the right opportunity for you. Our team is responsible for conducting global NPS studies to help business teams drive strategic initiatives that advocate for the best possible customer experience. We have a diverse suite of skill sets on the team, including Business Analysts, Business Intelligence Engineers, Data Scientists, Product Managers, and Software Development Engineers. Data Scientists in our team apply their skills to design studies and surveys, design and apply statistical analyses, and apply NLP-based processing and machine learning algorithms. Data Scientists also continuously learn new systems, tools, and industry best practices to analyze big data and enhance our analytics.
US, WA, Bellevue
ML scientists at Amazon participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and time-series forecasting applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.The AWS Demand Forecasting and Planning team is responsible for growing the world's largest Cloud. We forecast customer demand, build ML systems that understand customer needs, and drive utilization improvement for all AWS services.What we own:· Building a world-class forecasting platform that scales to handling billions of time series data in real time.· Developing predictive customer analytics models and recommendation engines.· Expanding inventory replenishment models and systems for each AWS service in the fast-growing AWS product portfolio.· Finding out the optimal tradeoff between AWS service availability and fleet utilization.· Driving fleet utilization improvement where each 1% means tens of millions of additional free cash flow.· Automating tactical and strategic capacity planning tools to optimize for service availability and infrastructure cost.What you will learn:· State of the art forecasting methodologies.· Application of machine learning to large-scale customer analytics.· Inventory management and supply chain management for the Cloud.· Resource management and admission control for the Cloud.· The internals of all AWS services.Keywords:Forecasting, Statistics, Machine Learning, Optimization, Inventory Management, Supply Chain Management, AWS, Cloud, Cloud Computing, EC2, S3, EBS, DynamoDB, CloudFront, Java, C++, Object Oriented, R, Distributed Systems, High Availability, Scalability, Concurrent
US, WA, Seattle
Amazon is focused on protecting the health and safety of our employees while continuing to serve people who need our services more than ever. Regular testing on a global scale across all industries would both help keep people safe and help get the economy back up and running. But, for this to work, we as a society would need vastly more testing capacity than is currently available. Unfortunately, today we live in a world of scarcity where COVID-19 testing is heavily rationed.To help solve this problem, Amazon has begun building incremental testing capacity and conducting research and development to apply cutting-edge technology advances to solve testing bottlenecks. We are seeking a Bioinformatics Lead to help define new approaches and execute against scaling them. You will work with a unique and gifted team developing these and collaborate with cross-functional teams. Our team rewards intellectual curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the cutting edge of both academic and applied research in this product area, you have the opportunity to work together with some of the most talented scientists, engineers, and product managers in the field.
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services, LLCTitle: Applied Scientist IILocation: Seattle, WAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services, LLCTitle: Data Scientist IILocation: Seattle, WAPosition Responsibilities:Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production.Amazon.com is an Equal Opportunity-Affirmative Action Employer - Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, MA, Cambridge
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services, LLCTitle: Applied Scientist IILocation: Cambridge, MAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and natural language (NL) applications. Develop and apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer - Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, AZ, Tempe
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services LLCTitle: Data Scientist IILocation: TEMPE, AZPosition Responsibilities:Own the data science elements of various Account Clustering Data Services products to help with data based decision making, product performance optimization, risk-based clustering, and product performance tracking. Work directly with product managers to help drive the design of the product. Work with Technical Product Managers to help drive the build planning. Translate business problems and products into data requirements and metrics. Initiate the design, development, and implementation of scientific analysis projects or deliverables. Own the analysis, modelling, system design, and development of data science solutions for products. Bridge the Degree of uncertainty in both problem definition and data scientific solution approaches. Build consensus on data, metrics, and analysis to drive business and system strategy.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
ES, B, Barcelona
Are you a Master’s or Ph.D. student interested in applying machine learning techniques to big-data sets? Are you excited by analyzing and modeling terabytes of text, images, and other types of data to solve real-world problems? We’re looking for smart scientists capable of using machine learning and statistical techniques to invent state-of-the-art solutions to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.Amazon has multiple positions available for Machine Learning Scientists in Barcelona.
US, CO, Boulder
Amazon Advertising operates at the intersection of eCommerce and advertising, offering a rich array of digital display advertising solutions with the goal of helping our customers find and discover anything they want to buy. We help advertisers reach Amazon customers on Amazon.com, across our other owned and operated sites, on other high quality sites across the web, and on millions of Kindles, tablets, and mobile devices. We start with the customer and work backwards in everything we do, including advertising. If you’re interested in joining a rapidly growing team working to build a unique, truly innovative advertising group with a relentless focus on the customer, you’ve come to the right place.Our team, Machine Learning Optimization, develops machine learning algorithms in high performance, petabyte-scale distributed systems. Our systems process billions of ad impressions daily from across the internet to power all of Amazon’s advertising reporting, as well as algorithms for audience targeting, real time ad ranking and bidding, and automated campaign optimization.We are looking for a talented Business Intelligence Engineer who is passionate about building metrics and pipelines and has the growth goal of getting deep in quantitative algorithm analysis. In this role, you will work with scientists, engineers, and product managers on high impact initiatives in Amazon’s Display Advertising.Job responsibilities:· Innovate new machine learning approaches for advertising targeting and optimization· Research and implement novel experimental design and measurement methodologies· Establish scalable, efficient, automated processes for large scale machine learning· Leverage petabyte scale data in strategic analysis for new monetization strategies, products and business directionsImpact and Career Growth:· Identify problems and opportunities leading to significant business impact.· Leverage petabyte scale data.· Opportunity to grow and broaden your technical skills as you work in an environment that thrives on creativity, experimentation, and product innovation.· Drive real-time algorithms to allocate billions of ads per day in advertising auctions.· Have the ability to experiment effectively with meaningful projects.Amazon is investing heavily in building a world class advertising business. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities. Together, we will change the face of advertising and retail.You can join our Seattle, New York or Boulder offices.
US, CA, Palo Alto
The Amazon Search team creates powerful, customer-focused search and advertising solutions and technologies. Whenever a customer visits an Amazon site worldwide and types in a query or browses through product categories, our Product Search services go to work. We design, develop, and deploy high performance, fault-tolerant distributed search systems used by millions of Amazon customers every day.Our Search Relevance team works to maximize the quality and effectiveness of the search experience for visitors to Amazon websites worldwide. The Strategic Relevance team builds solutions to improve search quality and effectiveness for Grocery and Media categories (Video, Music, Books, Aps etc). In this role you will:· Build machine learning models for Product Search.· Develop new ranking features and techniques building upon the latest results from the academic research community.· Propose and validate hypothesis to direct our business and product road map. Work with engineers to make low latency model predictions and scale the throughput of the system.· Focus on identifying and solving customer problems with simple and elegant solutions.· Design, develop, and implement production level code that serves billions of search requests. Own the full development cycle: design, development, impact assessment, A/B testing (including interpretation of results) and production deployment.· Collaborate with other engineers and related teams across Amazon to find technical solutions to complex design problems.Joining this team, you’ll experience the benefits of working in a dynamic, entrepreneurial environment, while leveraging the resources of Amazon.com (AMZN), one of the world's leading Internet companies. We provide a highly customer-centric, team-oriented environment in our offices located in Palo Alto, California.
US, WA, Seattle
Amazon created one of the most sophisticated supply chains in the world. From the introduction of Amazon Prime, to the use of advanced technology for package delivery, Amazon consistently drives change from the front of the pack.Amazon is seeking a detail oriented Data Scientist to focus on simulation to assist with process improvement and facility design initiatives in our North American fulfillment network. Successful candidates will be natural self-starters who have the drive to design, model, and simulate new fulfillment center conception and design processes.The Data Scientist will be expected to deep dive problems and drive relentlessly towards creative solutions. This individual needs to be comfortable interfacing and driving various functional teams and individuals at all levels of the organization in order to be successful. Perform data modelling using different discrete event simulation software’s, and use optimization techniques, research aptitude, and data analysis such as statistical analysis, regression analysis, Design of Experiments (DOE) etc. to drive decisions on process and designs.This role requires a strong passion for customers, a high level of comfort navigating ambiguity, and a keen sense of ownership and drive to deliver results. The ideal candidate will have experience in Science work, business analytics and have the aptitude to incorporate new approaches and methodologies while dealing with ambiguities in sourcing processes. Excellent business and communication skills are a must to develop and define key business questions and to build data sets. You should have demonstrated ability to think strategically and analytically about business, product, and technical challenges.You must be responsive, flexible, and able to succeed within an open collaborative environment. Amazon’s culture encourages innovation and expects to take a high-level of ownership in solving complex problems.Come help us make history!RESPONSIBILITIES:· Design, develop, and simulate engineering solutions for complex material handling challenges considering human/equipment interactions for the North America fulfillment network· Retrieve, synthesize, and present critical data in a format that is immediately useful to answering specific questions or improving system performance.· Analyze historical data to identify trends and support decision making.· Provide requirements to develop analytic capabilities, platforms, and pipelines.· Design, size, and analyze field experiments at scale.· Build decision-making models and propose solution for the business problem you defined. This may include delivery of algorithms to be used in production systems.· Conduct written and verbal presentation to share insights and recommendations to audiences of varying levels of technical sophistication.· Utilize code (python or another object oriented language) for data analyzing and modeling algorithm· Develop, document and update simulation standards, including standard results reports· Create basic to highly advanced models and run "what-if" scenarios to help drive to optimal solutions· Analyze historical data to identify trends and support decision making.· Apply statistical or machine learning knowledge to specific business problems and data.· Improve upon existing methodologies by developing new data sources, testing model enhancements, and fine-tuning model parameters.· Provide requirements to develop analytic capabilities, platforms, and pipeline· Ability to travel up to 10%
US, VA, Arlington
Ever wonder how you can apply your analytics skills to drive customer experience improvements for a diverse, innovative, and fun company like Amazon? Amazon’s Customer Insights and Research team is looking for a Data Scientist to help do exactly that and much more! If you have experience in surveys, experimental design, statistics and econometrics, and are interested in going deeper into data, this is the right opportunity for you. Our team is responsible for conducting global NPS studies to help business teams drive strategic initiatives that advocate for the best possible customer experience. We have a diverse suite of skill sets on the team, including Business Analysts, Business Intelligence Engineers, Data Scientists, Product Managers, and Software Development Engineers. Data Scientists on our team apply their skills to design studies and surveys, design and apply statistical analyses, and apply NLP-based process. Data Scientists also continuously learn new systems, tools, and industry best practices to analyze big data and enhance our analytics.