Applying PECOS to product retrieval and text autocompletion

Two KDD papers demonstrate the power and flexibility of Amazon’s framework for “extreme multilabel ranking”.

In April, our research team at Amazon open-sourced our PECOS framework for extreme multilabel ranking (XMR), which is the general problem of classifying an input when you have an enormous space of candidate classes. PECOS presents a way to solve XMR problems that is both accurate and efficient enough for real-time use.

At this year’s Knowledge Discovery and Data Mining Conference (KDD), members of our team presented two papers that demonstrate both the power and flexibility of the PECOS framework.

Retrieved products.png
A comparison of the top ten products returned by the PECOS-based product retrieval system and two predecessors for the query "rose of jericho plant". Products outlined in green were purchased by at least one customer performing that search; products outlined in red were not purchased.

One applies PECOS to the problem of product retrieval, a use case very familiar to customers at the Amazon Store. The other is a less obvious application: session-aware query autocompletion, in which an autocompletion model — which predicts what a customer is going to type — bases its predictions on the customer’s last few text inputs, as well as on statistics for customers at large.

In both cases, we tailor PECOS’s default models to the tasks at hand and, in comparisons with several strong benchmarks, show that PECOS offers the best combination of accuracy and speed.

The PECOS model

The classic case of XMR would be the classification of a document according to a handful of topics, where there are hundreds of thousands of topics to choose from.

We generalize the idea, however, to any problem that, for a given input, finds a few matches from among a large set of candidates. In product retrieval, for instance, the names of products would be “labels” we apply to a query: “Echo Dot”, “Echo Studio”, and other such names would be labels applied to the query “Smart speaker”.

PECOS adopts a three-step solution to the XMR problem. First is the indexing step, in which PECOS groups labels according to topic. Next is the matching step, which matches an input to a topic (which significantly shrinks the space of candidates). Last comes the ranking step, which reranks the labels in the matched topic, based on features of the input.

PECOS-framework.png
The three-stage PECOS model.
Credit: Stacy Reilly

PECOS comes with default models for each of these steps, which we described in a blog post about the April code release. But users can modify those models as necessary, or create their own and integrate them into the PECOS framework.

Product retrieval

For the product retrieval problem, we adapt one of the matching models that comes standard with PECOS: XR-Linear. Details are in the earlier blog post (and in our KDD paper), but XR-Linear reduces computation time by using B-ary trees — a generalization of binary trees to trees whose nodes have B descendants each. The top node of the tree represents the full label set; the next layer down represents B partitions of the full set; the next layer represents B partitions of each partition in the previous layer, and so on.

Connections between nodes of the trees have associated weights, which are multiplied by features of the input query to produce a probability score. Matching is the process of tracing the most-probable routes through the tree and retrieving the topics at the most-probable leaf nodes. To make this process efficient, we use beam search: i.e., at each layer, we limit the number of nodes whose descendants we consider, a limit known as the beam width.

Beam search.gif
An example of linear ranking with a beam width of two. At each level of the tree, two nodes (green) are selected for further exploration. Each of their descendant nodes is evaluated (orange), and two of those are selected for further exploration.
Credit: Giana Bucchino

In our KDD paper on product retrieval, we vary this general model through weight pruning; i.e., we delete edges whose weights fall below some threshold, reducing the number of options the matching algorithm has to consider as it explores the tree. In the paper, we report experiments with several different weight thresholds and beam widths.

We also experimented with several different sets of input features. One was n-grams of query words. For instance, the query “Echo with screen” would produce the 1-grams “Echo”, “with”, “screen”, the 2-grams “Echo with” and “with screen”, and the 3-gram “Echo with screen”. This sensitizes the matching model to phrases that may carry more information than their constituent words.

Similarly, we used n-grams of input characters. If we use the token “#” to denote the end of a word, the same query would produce the character trigrams “Ech”, “cho”, “ho#”, “with”, “ith”, and so on. Character n-grams helps the model deal with typos or word variants.

Finally, we also used TF-IDF (term frequency–inverse document frequency) features, which normalize the frequency of a word in a given text by its frequency across all texts (which filters out common words like “the”). We found that our model performed best when we used all three sets of features.

As benchmarks in our experiments, we used the state-of-the-art linear model and the state-of-the-art neural model and found that our linear approach outperformed both, with a recall@10 — that is, the number of correct labels among the top ten — that was more than double the neural model’s and almost quadruple the linear model’s. At the same time, our model took about one-sixth as long to train as the neural model.

We also found that our model took an average of only 1.25 milliseconds to complete each query, which is fast enough for deployment in a real-time system like the Amazon Store.

Session-aware query autocompletion

Session-aware query autocompletion uses the history of a customer’s recent queries — not just general statistics for the customer base — to complete new queries. The added contextual information means that it can often complete queries accurately after the customer has typed only one or two letters.

To frame this task as an XMR problem, we consider the case in which the input is a combination of the customer’s previous query and the beginning — perhaps just a few characters — of a new query. The labels are queries that an information retrieval system has seen before.

In this case, PECOS didn’t work well out of the box, and we deduced that the problem was the indexing scheme used to cluster labels by topic. PECOS’s default indexing model embeds inputs, or converts them into vectors, then clusters labels according to proximity in the vector space.

We suspected that this was ineffective when the inputs to the autocompletion model were partial phrases — fragments of words that a user is typing in. So we experimented with an indexing model that instead used data structures known as tries(a variation on “tree” that borrows part of the word “retrieve”).

A trie is a tree whose nodes represent strings of letters, where each descendant node extends its parent node’s string by one letter. So if the top node of the trie represents the letter “P”, its descendants might represent the strings “PA” and “PE”; their descendants might represent the strings “PAN”, “PAD”, “PEN”, “PET”, and so on. With a trie, all the nodes that descend from a common parent constitute a cluster.

Clustering using tries dramatically improved the performance of our model, but it also slowed it down: the strings encoded by tries can get very long, which means that tracing a path through the trie can get very time consuming.

So we adopted a hybrid clustering technique that combines tries with embeddings. The top few layers of the hybrid tree constitute a trie, but the nodes that descend from the lowest of these layers represent strings whose embeddings are near that of the parent node in the vector space.

Tree, Trie, Trie-tree hybrid.cloned.png
Three different ways of clustering the eight strings "a", "ab", "abc", "abd", "abfgh", "abfgi", "bcde", and "bcdf". At left is a conventional tree; in the center is a trie; and at right is a trie-tree hybrid.

To ensure that the embeddings in the hybrid tree preserve some of the sequential information encoded by tries, we varied the standard TF-IDF approach. First we applied it at the character level, rather than at the word level, so that it measured the relative frequency of particular strings of letters, not just words.

Then we weighted the frequency statistics, overcounting character strings that occurred at the beginning of words, relative to those that occurred later. This forced the embedding to mimic the string extension logic of the tries.

Once we’d adopted this indexing scheme, we found that the PECOS model outperformed both the state-of-the-art linear model and the state-of-the art neural model, when measured by both mean reciprocal rank and the BLEU metric used to evaluate machine translation models.

The use of tries still came with a performance penalty: our model took significantly longer to process inputs than the earlier linear model did. But its execution time was still below the threshold for real-time application and significantly lower than the neural model’s.

Related content

LU, Luxembourg
Have you ever wondered how Amazon delivers timely and reliably hundreds of millions of packages to customer’s doorsteps? Are you passionate about data and mathematics, and hope to impact the experience of millions of customers? Are you obsessed with designing simple algorithmic solutions to very challenging problems? If so, we look forward to hearing from you! Amazon Transportation Services is seeking Applied (or Research) Scientists. As a key member of the central Research Science Team of ATS operations, these persons will be responsible for designing algorithmic solutions based on data and mathematics for optimizing the middle-mile Amazon transportation network. The job is opened in the EU Headquarters in Luxembourg (alternatively: Barcelona, Berlin or London), designed to maximize interaction with the team and stakeholders, but we will consider applicants with remote work requirements as well. Key job responsibilities Solve complex optimization and machine learning problems using scalable algorithmic techniques. Design and develop efficient research prototypes that address real-world problems in the middle-mile operations of Amazon. Lead complex time-bound, long-term as well as ad-hoc analyses to assist decision making. Communicate to leadership results from business analysis, strategies and tactics. A day in the life You will be brainstorming algorithmic approaches with team-mates to solve challenging problems for the middle-mile operations of Amazon. You will be developing and testing prototype solutions with above algorithmic techniques. You will be scavenging information from the sea of Amazon data to improve these solutions. You will be meeting with other scientists, engineers, stakeholders and customers to enhance the solutions and get them adopted. About the team The Science and Tech team of ATS EU is looking for candidates who are looking to impact the world with their mathematical and data-driven skills. ATS stands for Amazon Transportation Service, we are the middle-mile planners: we carry the packages from the warehouses to the cities in a limited amount of time to enable the “Amazon experience”. As the core research team, we grow with ATS business to support decision making in an increasingly complex ecosystem of a data-driven supply chain and e-commerce giant. We schedule more than 1 million trucks with Amazon shipments annually; our algorithms are key to reducing CO2 emissions, protecting sites from being overwhelmed during peak days, and ensuring a smile on Amazon’s customer lips. Our mathematical algorithms provide confidence in leadership to invest in programs of several hundreds millions euros every year. Above all, we are having fun solving real-world problems, in real-world speed, while failing & learning along the way. We use modular algorithmic designs in the domain of combinatorial optimization, solving complicated generalizations of core OR problems with the right level of decomposition, employing parallelization and approximation algorithms. We use deep learning, bandits, and reinforcement learning to put data into the loop of decision making. We like to learn new techniques to surprise business stakeholders by making possible what they cannot anticipate. For this reason, we work closely with Amazon scholars and experts from Academic institutions. We code our prototypes to be production-ready We prefer provably optimal solutions than heuristics, though we settle for heuristics when performance dictates it. Overall, we appreciate the value of correct modeling. We are open to hiring candidates to work out of one of the following locations: Luxembourg, LUX
US, VA, Herndon
Do you love decomposing problems to develop machine learning (ML) products that impact millions of people around the world? Would you enjoy identifying, defining, and building ML software solutions that revolutionize how businesses operate? The Global Practice Organization in Professional Services at Amazon Web Services (AWS) is looking for a Software Development Engineer II to build, deliver, and maintain complex ML products that delight our customers and raise our performance bar. You’ll design fault-tolerant systems that run at massive scale as we continue to innovate best-in-class services and applications in the AWS Cloud. Key job responsibilities Our ML Engineers collaborate across diverse teams, projects, and environments to have a firsthand impact on our global customer base. You’ll bring a passion for the intersection of software development with generative AI and machine learning. You’ll also: - Solve complex technical problems, often ones not solved before, at every layer of the stack. - Design, implement, test, deploy and maintain innovative ML solutions to transform service performance, durability, cost, and security. - Build high-quality, highly available, always-on products. - Research implementations that deliver the best possible experiences for customers. A day in the life As you design and code solutions to help our team drive efficiencies in ML architecture, you’ll create metrics, implement automation and other improvements, and resolve the root cause of software defects. You’ll also: - Build high-impact ML solutions to deliver to our large customer base. - Participate in design discussions, code review, and communicate with internal and external stakeholders. - Work cross-functionally to help drive business solutions with your technical input. - Work in a startup-like development environment, where you’re always working on the most important stuff. About the team The Global Practice Organization for Analytics is a team inside the AWS Professional Services Organization. Our mission in the Global Practice Organization is to be at the forefront of defining machine learning domain strategy, and ensuring the scale of Professional Services' delivery. We define strategic initiatives, provide domain expertise, and oversee the development of high-quality, repeatable offerings that accelerate customer outcomes. Inclusive Team Culture Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life Balance Our team puts a high value on work-life harmony. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. As such, this is a customer facing role in a hybrid delivery model. Project engagements include remote delivery methods and onsite engagement that will include travel to customer locations as needed. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future. This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed. We are open to hiring candidates to work out of one of the following locations: Atlanta, GA, USA | Austin, TX, USA | Boston, MA, USA | Chicago, IL, USA | Herndon, VA, USA | Minneapolis, MN, USA | New York, NC, USA | San Diego, CA, USA | San Francisco, CA, USA | Seattle, WA, USA
US, MA, North Reading
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics We are open to hiring candidates to work out of one of the following locations: North Reading, MA, USA | Seattle, WA, USA | Westborough, MA, USA
CA, BC, Vancouver
Amazon Web Services (AWS) is building a world-class marketing organization that drives awareness and customer engagement with the goal of educating developers, IT and line-of-business professionals, startups, partners, and executive decision makers about AWS services and solutions, their benefits, and differentiation. As the central data and science organization in AWS Marketing, the Data: Science and Engineering (D:SE) team builds measurement products, AI/ML models for targeting, and self-service insights capabilities for AWS Marketing to drive better measurement and personalization, improve data access and analytical self-service, and empower strategic data-driven decisions. We work globally as a central team and establish standards, benchmarks, and best practices for use throughout AWS Marketing. We are looking for a Principal Data Scientist with deep expertise in scaling measurement science, content ranking and rapid experimentation at scale, with strong interest in building scalable solutions in partnership with our engineering organization. You will lead strategic measurement science initiatives across AWS Marketing & Sales ranging anywhere between recommender engines, scaling experimentation and measurement science, real-time inference, and cross-channel orchestration. You are an hands-on innovator who can contribute to advancing Marketing measurement technology in a B2B environment, and push the limits on what’s scientifically possible with a razor sharp focus on measurable customer and business impact. You will work with recognized B2B Marketing Science and AI/ML experts to develop large-scale, high-performing measurement science models and AI/ML capabilities. We are at a pivotal moment in our organization where AI/ML and measurement velocity has reached an unseen momentum, and we need to scale fast in order to maintain it. Your work will be a key input into a few of our key business goals. You will advance the state of the art in measurement at scale. We are open to hiring candidates to work out of one of the following locations: Vancouver, BC, CAN
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best. Key job responsibilities • Develop automated laboratory workflows. • Perform data QC, document results, and communicate to stakeholders. • Maintain updated understanding and knowledge of methods. • Identify and escalate equipment malfunctions; troubleshoot common errors. • Participate in the updating of protocols and database to accurately reflect the current practices. • Maintain equipment and instruments in good operating condition • Adapt to unexpected schedule changes and respond to emergency situations, as needed. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Seattle
Are you excited about developing generative AI and foundation models to revolutionize automation, robotics and computer vision? Are you looking for opportunities to build and deploy them on real problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience - at Amazon scale. We are looking for scientists, engineers and program managers for a variety of roles. The Amazon Robotics software team is seeking a Applied Scientist to focus on large vision and manipulation machine learning models. This includes building multi-viewpoint and time-series computer vision systems. It includes using machine learning to drive hardware movement. It includes building large-scale models using data from many different tasks and scenes. This work spans from basic research such as cross domain training, to experimenting on prototype in the lab, to running wide-scale A/B tests on robots in our facilities. Key job responsibilities * Research vision - Where should we be focusing our efforts * Research delivery – Proving/dis-proving strategies in offline data or in the lab * Production studies - Insights from production data or ad-hoc experimentation. About the team This team invents and runs robots focused on grasping and packing items. These are typically 6-dof style robotic arms. Our work ranges from the long-term-research on basic science to deploying/supporting large production fleets handling billions of items per year. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, VA, Arlington
Amazon launched the Generative AI (GenAI) Innovation Center (GAIIC) in Jun 2023 to help AWS customers accelerate enterprise innovation and success with Generative AI (https://press.aboutamazon.com/2023/6/aws-announces-generative-ai-innovation-center). Customers such as Highspot, Lonely Planet, Ryanair, and Twilio are engaging with the GAI Innovation Center to explore developing generative solutions. GAIIC provides opportunities to innovate in a fast-paced organization that contributes to game-changing projects and technologies that get deployed on devices and in the cloud. As a data scientist at GAIIC, you are proficient in designing and developing advanced Generative AI based solutions to solve diverse customer problems. You will be working with terabytes of text, images, and other types of data to solve real-world problems through Gen AI. You will be working closely with account teams and ML strategists to define the use case, and with other scientists and ML engineers on the team to design experiments, and find new ways to deliver value to the customer. The successful candidate will possess both technical and customer-facing skills that will allow you to be the technical “face” of AWS within our solution providers’ ecosystem/environment as well as directly to end customers. You will be able to drive discussions with senior technical and management personnel within customers and partners. This position requires that the candidate selected be a US Citizen and currently possess and maintain an active Top Secret security clearance. About the team Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Denver, CO, USA
US, CA, Santa Monica
Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities. We are looking for an Applied Scientist to join Monetization and Growth team in Marketplace Intelligence with a broad mandate to experiment and innovate to grow Sponsored Products. As an Applied Scientist in this team, you will help to identify unique opportunities to create customized and delightful shopping experience for our growing marketplaces worldwide. Your job will be identify big opportunities for the team that can help to grow Sponsored Products business working with retail partner teams, Product managers, Software engineers and TPMs. You will have opportunity to design, run and analyze A/B experiments to improve the experience of millions of Amazon shoppers while driving quantifiable revenue impact. More importantly, you will have the opportunity to broaden your technical skills in an environment that thrives on creativity, experimentation, and product innovation. Key job responsibilities - Be the technical leader in Machine Learning; lead efforts within this team and across other teams. - Perform hands-on analysis and modeling of enormous data sets to develop insights that increase traffic monetization and merchandise sales, without compromising the shopper experience. - Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. - Run A/B experiments, gather data, and perform statistical analysis. - Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. - Research new and innovative machine learning approaches. - Recruit Scientists to the team and provide mentorship. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Los Angeles, CA, USA | Santa Monica, CA, USA
US, VA, Arlington
Amazon’s mission is to be the most customer centric company in the world. The Workforce Staffing (WFS) organization is on the front line of that mission by hiring the hourly fulfillment associates who make that mission a reality. To drive the necessary growth and continued scale of Amazon’s associate needs within a constrained employment environment, Amazon has created the Workforce Intelligence (WFI) team. This team will (re)invent how Amazon attracts, communicates with, and ultimately hires its hourly associates. This team owns multi-layered research and program implementation to drive deep learning, process improvements, and strategic recommendations to global leadership. Are you passionate about data? Do you enjoy questioning the status quo? Do complex and difficult challenges excite you? If yes, this may be the team for you. The Data Scientist will be responsible for creating cutting edge algorithms, predictive and prescriptive models as well as required data models to facilitate WFS at-scale warehouse associate hiring. This role acts as an internal consultant to the marketing, biz ops and candidate experience teams covering responsibilities such as at-scale hiring process improvement, analyzing large scale candidate/associate data and being strategic to providing best candidate hiring experience to WFS warehouse associate candidates. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA
US, VA, Arlington
Amazon’s mission is to be the most customer centric company in the world. The vision of Workforce Intelligence is to design the ideal workforce to meet the customer promise anywhere. This organization leads and influences global workforce strategies that enable Amazon to scale operations more efficiently while also providing a unique voice for the hourly workforce. This is accomplished through a variety of science driven initiatives, experimentations, ML driven modeling, and data engineering. Amazon’s mission is to be the most customer-centric company in the world and we are on the front lines of that mission by providing robust research, data science and analytics to fill our jobs across the globe. You will identify data requirements, build methodology and tools that are statistically grounded. You will develop and produce actionable insights that allows our staffing teams to uncover opportunities to improve hiring. You will provide data-driven solutions that increase the efficiency of our hiring pipeline and improve candidate experience. Key responsibilities include: - As a Data Scientist (DS) in Workforce Intelligence, you will do causal data science, build predictive models, conduct simulations, create visualizations, and influence data science practice across the organization. - Provide insights by analyzing historical data from databases (Redshift, SQL Server, Oracle DW, and Salesforce). - Create experiments and prototype implementations of new learning algorithms and prediction techniques. - Research and build machine learning algorithms that improve hiring at scale. Inclusive Team Culture Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have 12 affinity groups (employee resource groups) with more than 87,000 employees across hundreds of chapters around the world. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which reminds team members to seek diverse perspectives, learn and be curious, and earn trust. Flexibility It isn’t about which hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We offer flexibility and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth We care about your career growth too. Whether your goals are to explore new technologies, take on bigger opportunities, or get to the next level, we'll help you get there. Our business is growing fast and our people will grow with it. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Seattle, WA, USA