A screen grab from an NFL video shows Packers quarterback Aaron Rodgers preparing to pass the ball
In January, the National Football League announced its new QB passing score, which addressed the inconsistency across plays, games, weeks, and seasons found in previous scores. A method based on spliced binned-Pareto distributions, developed by Amazon researchers, led to the improved passing metric.

The science behind NFL Next Gen Stats’ new passing metric

Spliced binned-Pareto distributions are flexible enough to handle symmetric, asymmetric, and multimodal distributions, offering a more consistent metric.

When football fans evaluate a player’s performance, they measure the player’s execution of specific plays against an innate sense of the player’s potential. Trying to encode such judgments into machine learning models, however, has proved non-trivial.

Fans and commentators have criticized existing quarterback (QB) passing stats, such as Madden QB, the NFL passer rating, ESPN’s total quarterback rating (QBR), and the Pro Football Focus (PFF) grade, for being calibrated to obsolete data, being unrelated to winning, or scoring players anomalously — as when Kyler Murray received the low Madden QB21 rating of 77 despite being the 2019 Offensive Rookie of the Year.

Related content
Principal data scientist Elena Ehrlich uses her skills to help a wide variety of customers — including the National Football League.

On January 13, 2022, just before Super Bowl LVI, the NFL announced its new QB passing score, which seeks to improve on its predecessors’ limitations and to isolate a QB’s contributions from those of the team in a completely data-driven way.

The play level

A root problem with existing ratings is their inconsistency across plays, games, weeks, and seasons. We sought a metric that could account for play-specific dynamics and scale to different granularities with consistency.

We wanted to measure the QB’s decision making and pass execution given the game clock and the pressure he was under. For those conditions, we have directly measurable quantities, such as the defense’s movements. But how do we measure how “well” the QB performed? This is a point we address in the next section (“The model architecture”), but for now, we take yards gained as a measurable outcome. (This assumption will prove useful downstream.)

nflendzonesideline.png
An (x, y)-coordinate representation of the football field.

Since we said we wanted to take a data-driven approach, let’s look at exactly what the data is.

On each play, we receive updates every 100 milliseconds from radio frequency ID chips in the players’ shoulder pads, giving us all 22 players’ position in the (x, y)-coordinates of the field, along with their speed, acceleration, running direction, and body orientation, as shown in the image above.

This time series is of variable length, starting with the snap and ending when the QB releases the ball. For example, a QB throwing four seconds after the snap yields a time series of 40 timesteps, whereas a pass that takes just over two seconds yields a time series of 25 timesteps.

Related content
In its collaboration with the NFL, AWS contributes cloud computing technology, machine learning services, business intelligence services — and, sometimes, the expertise of its scientists.

The figure below shows how the time series is represented. Each row corresponds to a single timestep and contains eight features (x-position, y-position, x-speed, y-speed, x-acceleration, y-acceleration, direction, and orientation) for each of 22 players, for a matrix of 176 columns and 40 rows. Features such as the number of defenders within a two-yard radius of the target receiver receive additional columns, but we eschew them here to focus on modeling technique.

nflplaytimeseriesmatrix.png
Matrix representation of the time series of a single play.

The collection of passing plays from the 2018-2020 seasons provided us with around 34,000 completions, 15,000 incompletes, and 1,200 interceptions, for more than 50,000 plays total. Feature preprocessing is a memory-intensive job, requiring two hours runtime on a ml.m5.m24xlarge instance. Modeling so large a number of time series, however, is a high-compute job.

For the model described in the upcoming section, the one-gpu p3.8xlarge instance incurred an eight-hour training time. While the NFL can afford two-hour preprocessing and eight-hour model fittings before the season commences, in live televised games, the inference returning a QB’s score for his play needs to be in real-time, like the 0.001 second per play of the following model.

The model architecture

To learn the temporal complexities within plays’ time series, we opted for a temporal convolutional network (TCN), a convolutional network adapted to handle inputs of different lengths and factor in long-range relationships between sequential inputs.

Since a play also has static attributes — such as down, score, and games remaining in the season — that influence players’ decisions and performance, we concatenate these with the TCN state and pass both to a multilayer perceptron to produce the final output, a probabilistic prediction of yards gained. To that, we compare the play’s actual yards gained.

nflplayertimeseriestcn.png
In our model, players’ time series are encoded by a temporal convolutional network (TCN), concatenated with a play’s static features, and fed to a multilayer perceptron.

Now, the network output is worth careful consideration. Naively, one might want to output a point prediction of the yards gained and train the network with an error loss function. But this fails to achieve the desired goal of measuring the outcome of a play relative to its potential.

An extra two yards gained under easier circumstances is not the same as two yards gained in more difficult circumstances, yet both would have a mean absolute error (MAE) of two yards. Instead, we opted for a distributional prediction, where the network’s outputs are parameters that specify a probability distribution.

We thought about which probability distribution function (PDF) would be most suitable. For certain plays, the PDF of yards gained would need to be asymmetrical: e.g., in a completed pass, if the QB throws to a receiver already running toward the end zone, positive yards gained are more likely than negative yards. Whereas for other plays, the PDF of yards gained would need to capture symmetry: on an interception, for example, the “negative” yards gained by the defender would balance against the possible positive yards gained by a completion.

There are even those plays for which the PDF would be bimodal: if the QB passes to a receiver with only one defender closing in, then the likelihood of yards gained lies either in the one- to two-yards range (if the receiver is tackled) or in the high-yardage range (if the receiver eludes the tackle), but not in-between. Other multi-model plays include when the QB may have to scramble for yards, like in the second play in this video.

yardsgainedpassescompletedgraphic.png
Yards gained on intercepted versus completed passes.

So we needed a distribution whose parameterization is flexible enough to accommodate multimodality, different symmetries, and light or heavy tails and whose locations and scale can vary with the clock time, current score, and other factors. We can’t meet these requirements with distributions like Gaussian or gamma, but we can meet them with the spliced binned-Pareto distribution.

The spliced binned-Pareto distribution

The spliced binned-Pareto (SBP) distribution arises from a classic result in extreme-value theory (EVT), which states that the distribution of extreme values (i.e., the tail) is almost independent of the base distribution of the data and, as shown below, can be estimated from the datapoints above the assumed upper bound (t) of the base distribution.

The second theorem of EVT states that any such distribution tail can be well-approximated by a generalized Pareto distribution (GPD) that has only two parameters, shape (x) and scale (b), and closed-form quantiles. The figure below shows the PDF of a GPD for x < 0, yielding a finite tail; x = 0, yielding an exponential tail; and x > 0, yielding a heavier-than-exponential tail.

valuesofdistribution.png
At left is a visualization of the observation that extreme values of a distribution (i.e., the tail) are almost independent of the base distribution and can be estimated from the datapoints above the assumed upper bound (t) of the base distribution. At right are probability distribution functions for generalized Pareto distributions with three different shapes.

Since we need multimodality and asymmetry for the base distribution, we modeled the base of the predictive distribution with a discrete binned distribution; as shown below, we discretize the real axis between two points into bins and predict the probability of the observation falling in each of these bins.

This yields a distribution robust to extreme values at training time because it is now a classification problem. The log-likelihood is not affected by the distance between the predicted mean and the observed point, as would be the case when using a Gaussian, Student’s t, or other parametric distribution. Moreover, the bins’ probability heights are independent of one another, so they can capture asymmetries or multiple modes in the distribution.

From the binned distribution, we delimit the lower tail by the fifth quantile and replace it with a weighted GPD. Analogously, we delimit the upper tail by the 95th quantile and replace it with another weighted GPD, to yield the SBP shown below.

binned and spliced binned graphic.png
At left is a binned distribution; at right is a spliced binned distribution, whose topmost and bottommost quantiles have been replaced with weighted generalized Pareto distributions.

The figure on the left above shows that the base distribution is indeed robust: the event represented by the extreme red dot will not bias the learned mean of the distribution but simply inflate the probability associated with the far-right bin.

However, this still leaves two problems: (i) although the red-dot event was observed to occur, the binned distribution would give it zero probability; conversely, (ii) the distribution would predict with certainty that extreme (i.e., great) plays do not occur. Because extreme yardage from deep-pass touchdowns, breakaway interceptions, etc., is rare, it is the adrenaline of the sport and exactly what we are most interested in describing probabilistically. The SBP figure above on the right graphically illustrates how the GPD tails can quantify how much less likely — i.e., harder — each incremental yard is.

The binned distribution and the GPDs are parameterized by the neural network we described above, which takes as input play matrices and outputs parameters: each of the bin probabilities, as well as x and b for each of the GPDs, which can be used to predict the probability-of-yards-gained value.

Establishing a gradient-based learning of heavy-tailed distributions has been a challenge in the ML community. Carreau and Bengio’s Hybrid Pareto model stitched GPD tails onto parametric distributions, but since the likelihood isn’t differentiable with respect to the threshold t, their model is supplemented with simulation and numerical approximations, foregoing time-varying applications. Other previous methods such as SPOT, DSPOT, and NN-SPOT, forego modeling the base and capture only the tails outside a fixed distance from the mean, which precludes higher-order non-stationarity and asymmetric tails.

While prior methods use a fixed threshold t to delimit tails, by modeling the base distribution, we obtain a time-varying threshold. Furthermore, training a single neural network to maximize the log-probability of the observed time step under the binned and GPD distributions yields a prediction that accounts for temporal variation in all moments of the distribution — the mean and variance as well as tail heaviness and scale, including asymmetric tails. The capabilities of different approaches are tabled below.

capabilitiesofdifferentapproaches.png
Capabilities of different approaches.

While we need a distributional prediction to grade a QB’s performance — to compare our model’s accuracy to other models’ — we need to use point predictions of yards gained. The table below compares the MAE of our method’s predictive median against that of a neural network with Gaussian output and against the point prediction of XGBoost, a decision-tree-based model.

meanaverageerror.png
Mean average error on yards gained for roughly 5,000 plays.

We have released Pytorch code for the spliced binned-Pareto model, along with a demo notebook.

The NGS passing score

Our model’s predictive PDF quantifies how likely each yardage gain is, for a league-average QB, given a specific play’s circumstances. Therefore, evaluating the actual yards gained in the cumulative distribution function (CDF) of that play’s SBP distribution yields a ranking between 0 and 1 of that QB’s performance relative to peer QBs.

This CDF ranking, under some further standardizations, becomes the QB passing score at the play level.

Aggregating scores over multiple plays yields game-, season-, or other split-level QB passing scores. For example, based on all targeted pass attempts in the ’21 season, Kyler Murray has a score of 87, ranking him ninth out of playoff QBs.

Under pressure, Murray's score jumps to 89; zooming in to passes between 2.5 and 4 seconds (in 2020 and 2021), Murray now scores a 99 in a five-way tie for the highest possible score. Other splits can also be contextualized with the NGS passing score, like deep passes, for example.

Finally, the tables below show that the NGS passing score correlates better with win percentages and playoff percentages than preceding passing metrics.

ngspassingscorespassingmetricsandwins.png
At left is the correlation of passing score with winning percentages and playoff percentages. At right is the comparison of passing score and other metrics.

Acknowledgments: Brad Gross

Research areas

Related content

GB, London
Have you ever ordered a product on Amazon and when that box with the smile arrived you wondered how it got to you so fast? Have you wondered where it came from and how much it cost Amazon to deliver it to you? We are looking for a Senior Data Scientist who will be responsible to develop cutting-edge scientific solutions to optimize our Pan-European fulfillment strategy, to maximize our Customer Experience and minimize our cost and carbon footprint. You will partner with the worldwide scientific community to help design the optimal fulfillment strategy for Amazon. You will also collaborate with technical teams to develop optimization tools for network flow planning and execution systems. Finally, you will also work with business and operational stakeholders to influence their strategy and gather inputs to solve problems. To be successful in the role, you will need deep analytical skills and a strong scientific background. The role also requires excellent communication skills, and an ability to influence across business functions at different levels. You will work in a fast-paced environment that requires you to be detail-oriented and comfortable in working with technical, business and technical teams. Key job responsibilities - Design and develop mathematical models to optimize inventory placement and product flows. - Design and develop statistical and optimization models for planning Supply Chain under uncertainty. - Manage several, high impact projects simultaneously. - Consult and collaborate with business and technical stakeholders across multiple teams to define new opportunities to optimize our Supply Chain. - Communicate data-driven insights and recommendations to diverse senior stakeholders through technical and/or business papers. We are open to hiring candidates to work out of one of the following locations: London, GBR
GB, London
Re-imagining the realms of what’s possible in advertising. Amazon is re-imagining advertising. Amazon Ads operates at the intersection of eCommerce and advertising and offering a rich array of advertising solutions and audience insights so businesses and brands can create relevant campaigns that produce measurable results. At Amazon Ads, you can build models that impact millions every day. And we’re passionate about solving real-world problems while using cutting-edge machine learning and artificial intelligence to do this. For example, our applied science teams leverage a variety of advanced machine learning and cloud computing techniques to power Amazon's advertising offerings. This includes building algorithms and cloud services using clustering, deep neural networks, and other ML approaches to make ads more relevant while respecting privacy. They develop machine learning models to predict ad outcomes and select the optimal ad for each shopper, context, and advertiser objective, leveraging techniques like multi-task learning, bandit/reinforcement learning, counterfactual estimation, and low-latency extreme ML. The teams also utilize Spark, EMR, and Elasticsearch to extract insights from big data and deliver recommendations to advertisers at scale, continuously improving through offline analysis and impact evaluation. Additionally, they apply generative AI models for dynamic creative optimization and video experimentation and automation. Underpinning these efforts are unique technical challenges, such as operating at unprecedented scale (hundreds of thousands of requests per second with 40ms latency) while respecting privacy and customer trust guarantees, and solving a wide variety of complex computational advertising problems related to traffic quality, viewability, brand safety, and more. Help us take innovation in advertising to the next level. Our teams are based in our fast-growing tech hubs in London and Edinburgh. Learn more about Amazon Ads, employee stories and available opportunities here: https://www.amazon.jobs/content/en/teams/advertising/applied-science-machine-learning-research?ref_=a20m_us_car_lp_asml Key job responsibilities * Design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both analysis and business judgment. * Collaborate with software engineering teams to integrate successful experiments into large-scale, highly complex Amazon production systems. * Promote the culture of experimentation and applied science at Amazon. * Demonstrate ability to meet deadlines while managing multiple projects. * Excel communication and presentation skills working with multiple peer groups and different levels of management * Influence and continuously improve a sustainable team culture that exemplifies Amazon’s leadership principles We are open to hiring candidates to work out of one of the following locations: Edinburgh, MLN, GBR | London, GBR
US, WA, Seattle
Amazon Shipping and Delivery Support (SDS) Tech team is seeking a passionate and customer-obsessed Senior Data Scientist to join our science team. You will use scientific research and rigorous analytics to influence our program and product strategies in driver and recipient support, solve complex problems at large scale, and drive intelligence and innovation in decision making. In this role, your main focus is to perform analysis, synthesize information, identify business opportunities, provide project direction, and communicate design and technical requirements within the team and across stakeholder groups. You will assist in defining trade-offs and quantifying opportunities for a variety of projects. You will learn current processes, build metrics, educate diverse stakeholder groups, assist product and tech teams in initial solution design, and audit new process flow implementations. Key job responsibilities * Provide thought leadership and support the development of continuously-evolving business analytics and data models, own the quantitative analysis of project opportunity and ROI. * Translate difficult business problem statements into data science frameworks; build, evaluate, and optimize statistical and machine learning models to solve focused business problems. * Retrieve, analyze, and synthesize critical data into a format that is immediately useful to answering specific questions or informing operational decisions. * Collaborate with product, program, and operations teams to design experiments (A/B Test) and analyze results to support launch decisions. * Conduct written and verbal presentations to share insights to audiences of varying levels of technical sophistication. A day in the life If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA | Seattle, WA, USA
US, VA, Arlington
We are seeking a Data Scientist to join our analytics team. This person will own the design and implementation of scalable and reliable approaches to support or automate decision making throughout the business. You will do this by analyzing data with a variety of statistical techniques and then building, validating, and implementing models based your analysis. You will not be able to do this alone but by building partnerships across data, engineering, and business teams. Key job responsibilities - Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult customer or business problems and cases in which the solution approach is unclear. - Proactively seek to identify business opportunities and insights and provide solutions to automate and optimize key internal and external products based on a broad and deep knowledge of Amazon data, industry best-practices, and work done by other teams. - Dive deep into the data and other models across the business to identify defects or inefficiencies which materially impact the customer or business, but can be mitigated through corrective actions for the AB Ops use case - Acquire this data by accessing data sources and building the necessary SQL/ETL queries or scripts. - Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. - Build models and automated tools using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. - Validate these models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. - Implement these models in a manner which complies with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production. - Enable product engineering teams to consume your models through services which can directly power customer-facing experiences. - Inspect the key business metrics/KPIs (even if you did not create them) when your analytics work points to potential gaps or opportunities; providing clear, compelling analyses by leveraging your knowledge across the AWS suite of products to support the broader business. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Seattle, WA, USA
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. About the team We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington and Palo Alto California. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers. We are open to hiring candidates to work out of one of the following locations: Palo Alto, CA, USA | Seattle, WA, USA
US, VA, Arlington
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The Generative AI Innovation Center at AWS is a new strategic team that helps AWS customers implement Generative AI solutions and realize transformational business opportunities. This is a team of strategists, data scientists, engineers, and solution architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and make plans for launching solutions at scale. The GenAI Innovation Center team provides guidance on best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for Data Scientists capable of using GenAI and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. A key focus of this role is GenAI model customization using techniques such as fine-tuning and continued pre-training to help customers build differentiating solutions with their unique data. Key job responsibilities As a Data Scientist, you will: Collaborate with AI/ML scientists and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms to address real-world challenges Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production Create and deliver best practice recommendations, tutorials, blog posts, sample code, and presentations adapted to technical, business, and executive stakeholder Provide customer and market feedback to Product and Engineering teams to help define product direction About the team Sales, Marketing and Global Services (SMGS) AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest-growing small- and mid-market accounts to enterprise-level customers, including the public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to customer success. AWS Support also partners with a global list of customers that are building mission-critical applications on top of AWS services. The Professional Services team is part of Global Services. About AWS Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Denver, CO, USA | Herndon, VA, USA | New York, NY, USA | Santa Clara, CA, USA | Seattle, WA, USA | Washington Dc, DC, USA
CN, 11, Beijing
Amazon Search JP builds features powering product search on the Amazon JP shopping site and expands the innovations to world wide. As an Applied Scientist on this growing team, you will take on a key role in improving the NLP and ranking capabilities of the Amazon product search service. Our ultimate goal is to help customers find the products they are searching for, and discover new products they would be interested in. We do so by developing NLP components that cover a wide range of languages and systems. As an Applied Scientist for Search JP, you will design, implement and deliver search features on Amazon site, helping millions of customers every day to find quickly what they are looking for. You will propose innovation in NLP and IR to build ML models trained on terabytes of product and traffic data, which are evaluated using both offline metrics as well as online metrics from A/B testing. You will then integrate these models into the production search engine that serves customers, closing the loop through data, modeling, application, and customer feedback. The chosen approaches for model architecture will balance business-defined performance metrics with the needs of millisecond response times. Key job responsibilities - Designing and implementing new features and machine learned models, including the application of state-of-art deep learning to solve search matching, ranking and Search suggestion problems. - Analyzing data and metrics relevant to the search experiences. - Working with teams worldwide on global projects. Your benefits include: - Working on a high-impact, high-visibility product, with your work improving the experience of millions of customers - The opportunity to use (and innovate) state-of-the-art ML methods to solve real-world problems with tangible customer impact - Being part of a growing team where you can influence the team's mission, direction, and how we achieve our goals We are open to hiring candidates to work out of one of the following locations: Beijing, 11, CHN | Shanghai, 31, CHN
GB, London
Re-imagining the realms of what’s possible in advertising. Amazon is re-imagining advertising. Amazon Ads operates at the intersection of eCommerce and advertising and offering a rich array of advertising solutions and audience insights so businesses and brands can create relevant campaigns that produce measurable results. At Amazon Ads, you can build models that impact millions every day. And we’re passionate about solving real-world problems while using cutting-edge machine learning and artificial intelligence to do this. For example, our applied science teams leverage a variety of advanced machine learning and cloud computing techniques to power Amazon's advertising offerings. This includes building algorithms and cloud services using clustering, deep neural networks, and other ML approaches to make ads more relevant while respecting privacy. They develop machine learning models to predict ad outcomes and select the optimal ad for each shopper, context, and advertiser objective, leveraging techniques like multi-task learning, bandit/reinforcement learning, counterfactual estimation, and low-latency extreme ML. The teams also utilize Spark, EMR, and Elasticsearch to extract insights from big data and deliver recommendations to advertisers at scale, continuously improving through offline analysis and impact evaluation. Additionally, they apply generative AI models for dynamic creative optimization and video experimentation and automation. Underpinning these efforts are unique technical challenges, such as operating at unprecedented scale (hundreds of thousands of requests per second with 40ms latency) while respecting privacy and customer trust guarantees, and solving a wide variety of complex computational advertising problems related to traffic quality, viewability, brand safety, and more. Help us take innovation in advertising to the next level. Our teams are based in our fast-growing tech hubs in London and Edinburgh. Learn more about Amazon Ads, employee stories and available opportunities here: https://www.amazon.jobs/content/en/teams/advertising/applied-science-machine-learning-research?ref_=a20m_us_car_lp_asml Key job responsibilities * Design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both analysis and business judgment. * Collaborate with software engineering teams to integrate successful experiments into large-scale, highly complex Amazon production systems. * Promote the culture of experimentation and applied science at Amazon. * Demonstrate ability to meet deadlines while managing multiple projects. * Excel communication and presentation skills working with multiple peer groups and different levels of management * Influence and continuously improve a sustainable team culture that exemplifies Amazon’s leadership principles * Develop a deep and wide understanding of large ad tech solutions to which you will contribute, and how they interact with components owned by other teams. * Anticipate obstacles and look around corners, effectively prioritising work, solving trade-offs and influencing the development of advertising products beyond the scope of your immediate team. We are open to hiring candidates to work out of one of the following locations: Edinburgh, MLN, GBR | London, GBR
US, VA, Arlington
Device Economics is looking for an economist experienced in causal inference, empirical industrial organization, forecasting, and scaled systems to work on business problems to advance critical resource allocation and pricing decisions in the Amazon Devices org. Output will be included in scaled systems to automate existing processes and to maximize business and customer objectives. Amazon Devices designs and builds Amazon first-party consumer electronics products to delight and engage customers. Amazon Devices represents a highly complex space with 100+ products across several product categories (e-readers [Kindle], tablets [Fire Tablets], smart speakers and audio assistants [Echo], wifi routers [eero], and video doorbells and cameras [Ring and Blink]), for sale both online and in offline retailers in several regions. The space becomes more complex with dynamic product offering with new product launches and new marketplace launches. The Device Economics team leads in analyzing these complex marketplace dynamics to enable science-driven decision making in the Devices org. Device Economics achieves this by combining economic expertise with macroeconomic trends, and including both in scientific applications for use by internal analysts, to provide deep understanding of customer preferences. Our team’s outputs inform product development decisions, investments in future product categories, product pricing and promotion, and bundling across complementary product lines. We have achieved substantial impact on the Devices business, and will achieve more. Device Economics seeks an economist adept in measuring customer preferences and behaviors with proven capacity to innovate, scale measurement, and drive rigor. The candidate will work with Amazon Devices science leadership to refine science roadmaps, models, and priorities for innovation and simplification, and advance adoption of insights to influence important resource allocation and prioritization decisions. Effective communication skills (verbal and written) are required to ensure success of this collaboration. The candidate must be passionate about advancing science for business and customer impact. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Seattle, WA, USA
US, TX, Austin
Passionate about helping customers simplify and accelerate their workloads on AWS? Are you fascinated by the power of Large Language Models (LLM), and applying ML and AI to solve complex challenges within one of Amazon's most significant businesses. Come join the AWS Marketplace team and help us build the future of partnerships and marketplace through ML. You will be a part of a team consisting of experienced Applied Scientists working on a new set of initiatives, building models and delivering them into the Amazon production ecosystem. Your efforts will build a robust ensemble of LLM artifacts and ML systems that can achieve high precision and recall, and scale to new marketplaces and languages. This problem is challenging due to sheer scale (billions of dollars of AWS revenue impact), diversity (hundreds of AWS services) and multitude of input sources (thousands of Partners and millions of Partner activity). We are looking for an experienced Scientist who can develop best in class solutions. Your primary customers are AWS customers and AWS Partners who would thank you for accelerating their migration to the AWS cloud across countries and industries. Inclusive Team Culture Our team is intentional about attracting, developing, and retaining amazing talent from diverse backgrounds. We’re looking for a new teammate who is enthusiastic, empathetic, curious, motivated, reliable, and able to work effectively with a diverse team of peers; someone who will help us amplify the positive & inclusive team culture we’ve been building. Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life Balance We understand that sometimes you need to work from home. Outside of quarantine, our team is co-located across our New York and Seattle offices. We work from home as needed for doctors appointments, to take our kids to school, pick them up, and do the things that can’t be done from the office. Mentorship & Career Growth We are dedicated to supporting new team members. Our team has a broad mix of experience levels and Amazon tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. You'll have an opportunity to learn from people with and without engineering and science backgrounds through one-on-one mentoring, and helpful code/design reviews. As a Senior Applied Scientist, you'll actively mentor others on the team on the art of what's possible with Machine Learning. Key job responsibilities The ideal Applied Scientist candidate has deep expertise in one or several of the following fields: Web search, Applied/Theoretical Machine Learning, Large Language Models, Deep Neural Networks, Classification Systems, Clustering, Label Propagation, Natural Language Processing, Computer Vision, Active learning, and Artificial Intelligence. S/he has a strong publication record at top relevant academic venues and experience in launching products/features in the industry. Do you want the excitement of experimenting with cutting edge Large Language Models (LLMs), machine learning, natural language processing, computer vision, and active learning models to solve real world problems at scale? Imagine experimenting with LLMs, with Deep Neural Networks as your daily job and imagine using your model outputs to affect the adoption of AWS cloud at scale through Partners. Imagine doing research inside of an Amazon team that is always looking to deploy creative solutions to real world problems in Cloud migration. Our scientists continue to publish, teach, and engage with the academic community, in addition to utilizing our working backwards method to enrich the way we live and work. About the team The AWS Partner Organization (APO) supports tens of thousands of consulting and technology partners who in turn serve AWS customers across nearly every industry, segment, and geography. The APO Engineering team is driven by the mission to provide best partner experience worldwide and create a better future for our customers and communities through a culture of Customer obsession, innovation and a relentless pursuit of excellence. Our team is responsible for both internal and external support for Partner Sales, the APN, and other key elements that drive the AWS partner engagement. Our commitment is to serve three primary stakeholder groups: the partners who collaborate with us to develop, promote, and sell AWS, the customers who rely on AWS services, and our internal users responsible for fostering AWS sales and partner-based adoption. This position is a critical enabler driving intelligence and science roadmaps for this entire organization. We are open to hiring candidates to work out of one of the following locations: Austin, TX, USA | Denver, CO, USA | New York City, NY, USA | Seattle, WA, USA