Adversarial training improves product discovery

Method automatically generates negative training examples for deep-learning model.

Deep learning has been highly successful for natural-language-processing tasks, like interpreting commands on Alexa. But it’s been less successful for information retrieval tasks, like product discovery on Amazon, due to a lack of negative training examples.

It’s relatively easy to teach a deep-learning model that, say, the query “Fire HD 10” matches the product named Fire HD 10. But teaching it that “Fire HD 10 case” and “Fire HD 10 charger” do not match the same product is more subtle and requires a lot of negative training examples. However, identifying and annotating negative examples for every item in a large product catalogue would be a massive undertaking.

This year, at the annual meeting of the Association for Computational Linguistics, my colleagues and I will present a new way to train deep-learning-based product discovery systems by automatically generating negative training examples. In our experiments, this approach improved performance by 16% over the state-of-the-art approach and by 62% over the approach typically used in commercial product discovery systems.

Our technique uses adversarial learning, which in recent years has proved remarkably successful in systems for generating visual images. In a typical adversarial-learning system, a generator that synthesizes data and a discriminator that tries to distinguish real data from synthetic are trained together.

In our case, however, the system is not trying to recognize synthetic examples; it’s simply trying to classify the synthetic examples accurately. This difference allowed us to design a new neural architecture that makes adversarial learning simpler and more efficient.

Match and mismatch

The inputs to our system are a customer query and a product name. The output is a decision about whether the product is a good match for the query.

For such a system, positive training examples are relatively easy to create automatically. Many shoppers who enter the search query “Fire HD 10” may in fact click on links for cases and chargers, but odds are that more will click on the link for the device itself. By aggregating statistics across many customer queries, an automated system can reliably produce examples of well-matched queries and products.

Negative examples are harder. Customers sometimes intentionally click mismatched items (such as cases and chargers after the query “Fire HD 10”), while conversely, a lack of clicks is no guarantee that queries and items are mismatched.

Hard examples.png
In this image, the black line indicates the boundary between products that do and do not match the query “running shoes”. Our goal was a generator that could produce negative examples (hiking shoe) within some small distance (dotted orange line) of the boundary.
Credit: Stacy Reilly

Existing natural-language-processing systems could determine that, for instance “telescope” is a bad match for the query “running shoes”, but a product discovery system wouldn’t learn much from such egregiously mismatched examples. Our goal: develop a generator that would automatically produce more challenging negative examples, such as the product “hiking shoes” mismatched with the query “running shoes”.

Hence our use of adversarial learning. During training, we feed our network automatically labeled positive examples. On a random basis, it chooses some of them for conversion to negative examples. The generator overwrites half of the example — the query — and changes its label from “match” to “mismatch”.

In a typical application of adversarial learning, the generator and discriminator are simultaneously trained on competing objectives, which complicates the machine learning process. In our case, however, we simply switch back and forth between objectives during training.

Quarts-model.png
The architecture of our model. The inputs are a query (Q) and an item (I). During training, the variational generator sometimes overwrites the query with a mismatched query intended to deceive the classifier. The mechanism for determining when to overwrite queries involves a set of simple arithmetic operations (orange bar).

Moreover, the switch mechanism is a set of simple arithmetic operations performed within the network, which means that we can train the network using the standard machine learning algorithm (back-propagation).

Another key to our network is the attention layer that immediately precedes the classification layer. The attention layer learns to concentrate the classifier’s attention on elements of both the query and the product name that are particularly important for assessing matches.

For instance, if the query is “Fire HD 10 case”, and the matched product is a Fire HD 10 case, the attention layer will give greater weight to the word “case” than to the word “Fire”, since “case” better distinguishes the query from other queries relating to Fire tablets.

Quarts-attention matrix.png
A diagram of the weights our attention layer assigned to the terms (row labels) of a query for each term (column labels) in a product description. Lighter colors indicate higher weights, as per the legend at right.

In experiments, we compared our model to four others, including gradient-boosted decision trees, which are commonly used in the product discovery space, and MatchPyramid, a model proposed four years ago that has been shown to significantly outperform other models on matching tasks.

We measured performance using two different metrics, both of which factor in both false positives and false negatives: F1 scores and area under the precision-recall curve (APR). MatchPyramid was the best-performing of the baselines, but our model surpassed it by 16% on F1 score and 8% on APR. Compared to gradient-boosted decision trees, our model’s improvements were 62% on F1 score and 57% on APR.

Related content

US, CA, Santa Clara
Job summaryAmazon is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background to help build industry-leading language technology.Our mission is to provide a delightful experience to Amazon’s customers by pushing the envelope in Natural Language Processing (NLP), Natural Language Understanding (NLU), Dialog management, conversational AI and Machine Learning (ML).As part of our AI team in Amazon AWS, you will work alongside internationally recognized experts to develop novel algorithms and modeling techniques to advance the state-of-the-art in human language technology. Your work will directly impact millions of our customers in the form of products and services, as well as contributing to the wider research community. You will gain hands on experience with Amazon’s heterogeneous text and structured data sources, and large-scale computing resources to accelerate advances in language understanding.We are hiring primarily in Conversational AI / Dialog System Development areas: NLP, NLU, Dialog Management, NLG.This role can be based in NYC, Seattle or Palo Alto.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, NY, New York
Job summaryAmazon is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background to help build industry-leading language technology.Our mission is to provide a delightful experience to Amazon’s customers by pushing the envelope in Natural Language Processing (NLP), Natural Language Understanding (NLU), Dialog management, conversational AI and Machine Learning (ML).As part of our AI team in Amazon AWS, you will work alongside internationally recognized experts to develop novel algorithms and modeling techniques to advance the state-of-the-art in human language technology. Your work will directly impact millions of our customers in the form of products and services, as well as contributing to the wider research community. You will gain hands on experience with Amazon’s heterogeneous text and structured data sources, and large-scale computing resources to accelerate advances in language understanding.We are hiring primarily in Conversational AI / Dialog System Development areas: NLP, NLU, Dialog Management, NLG.This role can be based in NYC, Seattle or Palo Alto.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, CA, Santa Clara
Job summaryAWS AI/ML is looking for world class scientists and engineers to join its AI Research and Education group working on building automated ML solutions for planetary-scale sustainability and geospatial applications. Our team's mission is to develop ready-to-use and automated solutions that solve important sustainability and geospatial problems. We live in a time wherein geospatial data, such as climate, agricultural crop yield, weather, landcover, etc., has become ubiquitous. Cloud computing has made it easy to gather and process the data that describes the earth system and are generated by satellites, mobile devices, and IoT devices. Our vision is to bring the best ML/AI algorithms to solve practical environmental and sustainability-related R&D problems at scale. Building these solutions require a solid foundation in machine learning infrastructure and deep learning technologies. The team specializes in developing popular open source software libraries like AutoGluon, GluonCV, GluonNLP, DGL, Apache/MXNet (incubating). Our strategy is to bring the best of ML based automation to the geospatial and sustainability area.We are seeking an experienced Applied Scientist for the team. This is a role that combines science knowledge (around machine learning, computer vision, earth science), technical strength, and product focus. It will be your job to develop ML system and solutions and work closely with the engineering team to ship them to our customers. You will interact closely with our customers and with the academic and research communities. You will be at the heart of a growing and exciting focus area for AWS and work with other acclaimed engineers and world famous scientists. You are also expected to work closely with other applied scientists and demonstrate Amazon Leadership Principles (https://www.amazon.jobs/en/principles). Strong technical skills and experience with machine learning and computer vision are required. Experience working with earth science, mapping, and geospatial data is a plus. Our customers are extremely technical and the solutions we build for them are strongly coupled to technical feasibility.About the teamInclusive Team CultureAt AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded scientist and enable them to take on more complex tasks in the future.Interested in this role? Reach out to the recruiting team with questions or apply directly via amazon.jobs.
US, CA, Santa Clara
Job summaryAWS AI/ML is looking for world class scientists and engineers to join its AI Research and Education group working on building automated ML solutions for planetary-scale sustainability and geospatial applications. Our team's mission is to develop ready-to-use and automated solutions that solve important sustainability and geospatial problems. We live in a time wherein geospatial data, such as climate, agricultural crop yield, weather, landcover, etc., has become ubiquitous. Cloud computing has made it easy to gather and process the data that describes the earth system and are generated by satellites, mobile devices, and IoT devices. Our vision is to bring the best ML/AI algorithms to solve practical environmental and sustainability-related R&D problems at scale. Building these solutions require a solid foundation in machine learning infrastructure and deep learning technologies. The team specializes in developing popular open source software libraries like AutoGluon, GluonCV, GluonNLP, DGL, Apache/MXNet (incubating). Our strategy is to bring the best of ML based automation to the geospatial and sustainability area.We are seeking an experienced Applied Scientist for the team. This is a role that combines science knowledge (around machine learning, computer vision, earth science), technical strength, and product focus. It will be your job to develop ML system and solutions and work closely with the engineering team to ship them to our customers. You will interact closely with our customers and with the academic and research communities. You will be at the heart of a growing and exciting focus area for AWS and work with other acclaimed engineers and world famous scientists. You are also expected to work closely with other applied scientists and demonstrate Amazon Leadership Principles (https://www.amazon.jobs/en/principles). Strong technical skills and experience with machine learning and computer vision are required. Experience working with earth science, mapping, and geospatial data is a plus. Our customers are extremely technical and the solutions we build for them are strongly coupled to technical feasibility.About the teamInclusive Team CultureAt AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded scientist and enable them to take on more complex tasks in the future.Interested in this role? Reach out to the recruiting team with questions or apply directly via amazon.jobs.
US, NY, New York
Job summaryAmazon Web Services is looking for world class scientists to join the Security Analytics and AI Research team within AWS Security Services. This group is entrusted with researching and developing core data mining and machine learning algorithms for various AWS security services like GuardDuty (https://aws.amazon.com/guardduty/) and Macie (https://aws.amazon.com/macie/). In this group, you will invent and implement innovative solutions for never-before-solved problems. If you have passion for security and experience with large scale machine learning problems, this will be an exciting opportunity.The AWS Security Services team builds technologies that help customers strengthen their security posture and better meet security requirements in the AWS Cloud. The team interacts with security researchers to codify our own learnings and best practices and make them available for customers. We are building massively scalable and globally distributed security systems to power next generation services.Inclusive Team Culture Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop and enable them to take on more complex tasks in the future.A day in the lifeAbout the hiring groupJob responsibilities* Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgment.* Collaborate with software engineering teams to integrate successful experiments into large scale, highly complex production services.* Report results in a scientifically rigorous way.* Interact with security engineers, product managers and related domain experts to dive deep into the types of challenges that we need innovative solutions for.
US, WA, Seattle
Job summaryWe are seeking a talented Economist to join our team and help us build innovative systems at the intersection of machine learning and quantitative marketing. We develop new measurement and optimization tools that enable Amazon to make smart marketing investment decisions. This role will impact billions of dollars of decision-making by Amazon’s most strategic businesses. Our most effective tools are released to external advertisers too, defining new industry standards. What makes us unique is our comprehensive data, our world-class engineering systems and a high concentration of some of the most talented scientists and engineers in industry. As a successful candidate, you will be passionate about building scalable systems. You'll be comfortable with ambiguity and have exceptional technical acumen. You'll be up to speed on the latest research, and capable of developing new techniques at the intersection of causal inference, reinforcement learning and quantitative marketing. You will lead other scientists by example, with crisp technical writing and frequent presentations.Key job responsibilitiesBuild end-to-end causal machine learning solutions.Perform hands-on analysis and modeling with enormous data sets to better understand how advertising influences shopper behavior.Run A/B experiments that affect millions of customers to evaluate the impact of your solutions.Spearhead new research agendas and own delivering on commitments.Work closely with engineering to design systems that facilitate high velocity science exploration and quick prototype-to-prod timelines.Develop novel methods at the intersection of causal inference, machine learning and quantitative marketing.Present original research at internal and external conferences.About the teamOur team is a dynamic mix of scientists who are passionate about innovating. We are excited to evaluate our novel causal models and optimization algorithms against ground truth generated by large-scale experiments. We think big, take risks and stay grounded.
IN, KA, Bangalore
Job summaryDo you want to be part of a new team at Amazon that is making history? Do you want to build technology and new science that millions of people will use? Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)?We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales.We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Bangalore (India). We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.Do not hesitate to reach out if you have some of the following: ability to apply state of the art in large scale Machine Learning (e.g. semi-weakly-un-supervised deep learning, natural language understanding), curiosity to learn through controlled experimentation or experience with low latency production systems. Apply now or ping Shailendra Agarwal (https://www.linkedin.com/in/shailendra-agarwal-893a3b11/) to learn more about the different ways you can have huge impact with us.We are an inclusive employer and value diversity at Amazon. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
US, VA, Virtual Location - Virginia
Job summaryJob summaryMachine learning (ML) has been strategic to Amazon from the early years. We are pioneers in areas such as recommendation engines, product search, eCommerce fraud detection, and large-scale optimization of fulfillment center operations.The Amazon ML Solutions Lab team helps AWS customers accelerate the use of machine learning to solve business and operational challenges and promote innovation in their organization. We are looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background to help develop solutions by pushing the envelope in Time Series, Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), Machine Learning (ML), Computer Vision (CV) and More.As a ML Solutions Lab Applied Scientist, you are proficient in designing and developing advanced ML models to solve diverse challenges and opportunities. You will be working with terabytes of text, images, and other types of data and develop novel models to solve real-world problems. You'll design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. You will apply classical ML algorithms and cutting-edge deep learning (DL) and reinforcement learning approaches to areas such as drug discovery, customer segmentation, fraud prevention, capacity planning, predictive maintenance, pricing optimization, call center analytics, player pose estimation, event detection, and virtual assistant among others.Key job responsibilitiesThe primary responsibilities of this role are to:Design, develop, and evaluate innovative ML/DL models to solve diverse challenges and opportunities across industriesInteract with customer directly to understand their business problems, and help them with defining and implementing scalable ML/DL solutions to solve themWork closely with account teams, research scientist teams, and product engineering teams to drive model implementations and new algorithmsThis position requires travel of up to 25%. Role can be preferably based in DC, Maryland, Virginia / New York, New Jersey area; However candidates in Boston or Atlanta area are encouraged to apply.
FR
Job summaryThe ER Solutions team uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which solve for defects in the Employee Experience. We are an interdisciplinary team which combines the talents of science, psychology and industry specialists to develop and deliver solutions that measurably achieve this goal. We are looking for an Economist who is able to provide structure around complex business problems, hone those complex problems into specific, scientific questions, and test those questions to generate insights. The ideal candidate will work with various science, PXT, ER, operations and tech teams to estimate models and algorithms on large scale data, design pilots and measure their impact, and transform successful prototypes into improved policies and programs at scale. Ideal candidates will work closely with business partners to develop science that solves the most important business challenges. They will need to work well in a team setting with individuals from diverse disciplines and backgrounds. Serve as an ambassador for science and a scientific resource for business teams, so that scientific processes permeate throughout the PXT organization to the benefit of Amazonians and Amazon. Ideal candidates will own the development of scientific models and manage the data analysis, modeling, and experimentation that is necessary for estimating and validating model. They will be customer-centric – clearly communicating scientific approaches and findings to business leaders, listening to and incorporate their feedback, and delivering successful scientific solutions.
US, WA, Seattle
Job summaryAmazon is looking for an outstanding Data Scientist to join the AWS Product Analytics and Data Science team. This is your opportunity to be a core member of the AWS Product Analytics team that has direct impact on the long-term roadmap of the AWS Product teams. This role is within a broader Data science, Business Intelligence & Data engineering team that focuses on using statistical modeling and machine learning to drive actionable and impactful business decisions across AWS. Since early 2006, AWS has provided companies of all sizes with an infrastructure platform in the cloud. AWS is a high-growth, fast-moving division within Amazon with a start-up mentality where new and diverse challenges arise every day.On the AWS Product Analytics team, you will be surrounded by people who are exceptionally talented, bright, and driven, and believe that data-driven decision making is critical to our success. To be successful in this role, you have a strong passion for analytics and accountability, set high standards with a focus on superior business outcome. You should also have strong business acumen who feels comfortable tackling ambiguous business problems in dynamic business environment. Your decision will influence AWS VP and Director level product and business decisions that directly impact AWS’ product roadmap and customer experience.We take working hard, having fun, and making history seriously. AWS sets the standard for functionality, cost, and performance for many cloud-based services, but it’s still early days for cloud computing, and there are boundless opportunities to continue to redefine the world of cloud computing - come help us make history!Key job responsibilitiesThe successful candidate will have a strong quantitative background and can thrive in an environment that leverages statistics, machine learning, operations research, econometrics, and strong business acumen. As a Data Scientist, you will discover and solve real world problems by analyzing one of the world’s largest datasets, developing statistical and machine learning models to drive business decisions, adopting best practices, and conducting data science research and development. You will also collaborate closely with business leaders, software engineers, economists, and researchers.You will adopt the best practices for delivering high quality data science projects, influencing analytics roadmap, setting best practicesYou will work on high visibility and high business impact problems that directly influence AWS product roadmaps and business decisions. You will spend time formulating and defining science problem based on business requirements.You will translate business problems into analytical framework and form testing hypotheses that can be answered with available data using scientific methods or identify additional data neededYou will work on a diverse set of analytics problems, such as user growth, pricing, forecasting, causal inference, marketing research, experimentation, and other machine learning problemsYou will collaborate with cross-function teams with AWS business leaders, data engineers, software engineers, economists, and other scientistsYou will help define key performance indicators for key business programs and propose analytics framework that drive business valueYou are familiar with state-of-the-art statistical and machine learning methods, understand the full life-cycle of analytics projects, have hands-on experience building end-to-end analytics solutionsInclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.This position involves on-call responsibilities, typically for one week every two months. We don’t like getting paged in the middle of the night or on the weekend, so we work to ensure that our systems are fault tolerant. When we do get paged, we work together to resolve the root cause so that we don’t get paged for the same issue twice.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.A day in the lifeYou will work directly with internal stakeholders including, product managers, go-to-market leaders, and other business leaders at manager and director level. You will work with customer face-to-face by decompose their ambiguous business problem into a science problem. You will adopt best practices and hold high standard for the team. You will test different business hypotheses and build statistical, machine learning, and causal inference models. You will solve the most impactful and challenging business today to provide actionable insights.About the teamWe are trusted analytics thought partner to enable AWS product and business leaders to make informed product decisions and drive revenue growth, through rigorous science methods, reusable analytics frameworks, and scalable analytics products.