Invalidating robotic ad clicks in real time

Slice-level detection of robots (SLIDR) uses deep-learning and optimization techniques to ensure that advertisers aren’t charged for robotic or fraudulent ad clicks.

Robotic-ad-click detection is the task of determining whether an ad click on an e-commerce website was initiated by a human or a software agent. Its goal is to ensure that advertisers’ campaigns are not billed for robotic activity and that human clicks are not invalidated. It must act in real time, to cause minimal disruption to the advertiser experience, and it must be scalable, comprehensive, precise, and able to respond rapidly to changing traffic patterns.

At this year’s Conference on Innovative Applications of Artificial Intelligence (IAAI) — part of AAAI, the annual meeting of the Association for the Advancement of Artificial Intelligence — we presented SLIDR, or SLIce-Level Detection of Robots, a real-time deep-neural-network model trained with weak supervision to identify invalid clicks on online ads. SLIDR has been deployed on Amazon since 2021, safeguarding advertiser campaigns against robotic clicks.

Related content
Paper introduces a unified view of the learning-to-bid problem and presents AuctionGym, a simulation environment that enables reproducible validation of new solutions.

In the paper, we formulate a convex optimization problem that enables SLIDR to achieve optimal performance on individual traffic slices, with a budget of overall false positives. We also describe our system design, which enables continuous offline retraining and large-scale real-time inference, and we share some of the important lessons we’ve learned from deploying SLIDR, including the use of guardrails to prevent updates of anomalous models and disaster recovery mechanisms to mitigate or correct decisions made by a faulty model.

Challenges

Detecting robotic activity in online advertising faces various challenges: (1) precise ground-truth labels with high coverage are hard to come by; (2) bot behavior patterns are continuously evolving; (3) bot behavior patterns vary significantly across different traffic slices (e.g., desktop vs, mobile); and (4) false positives reduce ad revenue.

Labels

Since accurate ground truth is unavailable at scale, we generate data labels by identifying two high-hurdle activities that are very unlikely to be performed by a bot: (1) ad clicks that lead to purchases and (2) ad clicks from customer accounts with high RFM scores. RFM scores represent the recency (R), frequency (F), and monetary (M) value of customers’ purchasing patterns on Amazon. Clicks of either sort are labeled as human; all remaining clicks are marked as non-human.

Metrics

Due to the lack of reliable ground truth labels, typical metrics such as accuracy cannot be used to evaluate the model performance. So we turn to a trio of more-specific metrics.

Related content
Amazon VP and chief economist for digital streaming and advertising Phil Leslie on economists’ role in industry.

Invalidation rate (IVR) is defined as the fraction of total clicks marked as robotic by the algorithm. IVR is indicative of the recall of our model, since a model with a higher IVR is more likely to invalidate robotic clicks.

On its own, however, IVR can be misleading, since a poorly performing model will invalidate human and robot clicks. Hence we measure IVR in conjunction with the false-positive rate (FPR). We consider purchasing clicks as a proxy for the distribution of human clicks and define FPR as the fraction of purchasing clicks invalidated by the algorithm. Here, we make two assumptions: (1) all purchasing clicks are human, and (2) purchasing clicks are a representative sample of all human clicks.

We also define a more precise variant of recall by checking the model’s coverage over a heuristic that identifies clicks with a high likelihood to be robotic. The heuristic labels all clicks in user sessions with more than k ad clicks in an hour as robotic. We call this metric robotic coverage.

A neural model for detecting bots

We consider various input features for our model that will enable it to disambiguate robotic and human behavior:

  1. User-level frequency and velocity counters compute volumes and rates of clicks from users over various time periods. These enable identification of emergent robotic attacks that involve sudden bursts of clicks.
  2. User entity counters keep track of statistics such as number of distinct sessions or users from an IP. These features help to identify IP addresses that may be gateways with many users behind them.
  3. Time of click tracks hour of day and day of week, which are mapped to a unit circle. Although human activity follows diurnal and weekly activity patterns, robotic activity often does not.
  4. Logged-in status differentiates between customers and non-logged-in sessions as we expect a lot more robotic traffic in the latter.

The neural network is a binary classifier consisting of three fully connected layers with ReLU activations and L2 regularization in the intermediate layers.

DNN architecture.png
Neural-network architecture.

While training our model, we use sample weights that weigh clicks equivalently across hour of day, day of the week, logged-in status, and the label value. We have found sample weights to be crucial in improving the model’s performance and stability, especially with respect to sparse data slices such as night hours.

Baseline comparison.png
Baseline comparison.

We compare our model against baselines such as logistic regression and a heuristic rule that computes velocity scores of clicks. Both the baselines lack the ability to model complex patterns and hence are unable to perform as well as the neural network.

Calibration

Calibration involves choosing a threshold for the model’s output probability above which all clicks are marked as invalid. The model should invalidate certain highly robotic clicks but at the same time not incur high revenue loss by invalidating human clicks. Toward this, one option is to pick the “knee” of the IVR-FPR curve, beyond which the false positive rate increases sharply when compared to the increase in IVR.

Full traffic.png
IVR-FPR curve of full traffic.

But calibrating the model across all traffic slices together leads to different behaviors for different slices. For example, a decision threshold obtained via overall calibration, when applied to the desktop slice, could be undercalibrated: a lower probability threshold could invalidate more bots. Similarly, when the global decision threshold is applied to the mobile slice, it could be overcalibrated: a higher probability threshold might be able to recover some revenue loss without compromising on the bot coverage.

To ensure fairness across all traffic slices, we formulate calibration as a convex optimization problem. We perform joint optimization across all slices by fixing an overall FPR budget (an upper limit to the FPR of all slices combined) and solve to maximize the combined IVR on all slices together. The optimization must meet two conditions: (1) each slice has a minimum robotic coverage, which establishes a lower found for its FPR, and (2) the combined FPR of all slices should not exceed the FPR budget.

Traffic slices.png
IVR-FPR curve of traffic slices.

Since the IVR-FPR curve of each slice can be approximated as a quadratic function of the FPR, solving the joint optimization problem finds appropriate values for each slice. We have found slice-level calibration to be crucial in lowering overall FPR and increasing robotic coverage.

Deployment

To quickly adapt to changing bot patterns, we built an offline system that retrains and recalibrates the model on a daily basis. For incoming traffic requests, the real-time component computes the feature values using a combination of Redis and read-only DB caches and runs the neural-network inference on a horizontally scalable fleet of GPU instances. To meet the real-time constraint, the entire inference service, which runs on AWS, has a p99.9 latency below five milliseconds.

SLIDR architecture 16x9.png
The SLIDR system design.

To address data and model anomalies during retraining and recalibration, we put certain guardrails on the input training data and the model performance. For example, when purchase labels are missing for a few hours, the model can learn to invalidate a large amount of traffic. Guardrails such as minimum human density in every hour of a week prevent such behavior.

Related content
Expo cochair and Amazon scientist Alice Zheng on the respective strengths of industry and academic machine learning research.

We have also developed disaster recovery mechanisms such as quick rollbacks to a previously stable model when a sharp metric deviation is observed and a replay tool that can replay traffic through a previously stable model or recompute real-time features and publish delayed decisions, which help prevent high-impact events.

In the future, we plan to add more features to the model, such as learned representations for users, IPs, UserAgents, and search queries. We presented our initial work in that direction in our NeurIPS 2022 paper, “Self supervised pre-training for large scale tabular data”. We also plan to experiment with advanced neural architectures such as deep and cross-networks, which can effectively capture feature interactions in tabular data.

Acknowledgements: Muneeb Ahmed

Related content

IN, KA, Bangalore
AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest growing small- and mid-market accounts to enterprise-level customers including public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to customer success. AWS Support also partners with a global list of customers that are building mission-critical applications on top of AWS services. Do you have proven analytical capabilities to identify business opportunities, develop predictive models and optimization algorithms to help us build state of the art Support organization? At Amazon, we are working to be the most customer-centric company on earth. To get there, we need exceptionally talented, bright, and driven people. We set big goals and are looking for people who can help us reach and exceed them. Amazon Web Services (AWS) is one of the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Amazon Web Services, Inc. provides services for broad range of applications including compute, storage, databases, networking, analytics, machine learning and artificial intelligence (AI), Internet of Things (IoT), security, and application development, deployment, and management. Global AWS Support BizOPs team is looking for a strong, talented Data Scientist to model contact forecasting, discovering insights and identifying opportunities through the use of statistics, machine learning, and deep learning to drive business and operational improvements. A successful candidate must be passionate about building solutions that will help drive a more efficient operations network and optimize cost. In this role, you will partner with data engineering, Tooling team, operations, Training, Customer Service, Capacity planning and finance teams, driving optimization and prediction solutions across the network. Key job responsibilities We are looking for an experienced and motivated Data Scientist with proven abilities to build and manage modeling projects, identify data requirements, build methodology and tools that are statistically grounded The candidate will be an expert in the areas of data science, optimization, machine learning and statistics, and is comfortable facilitating ideation and working from concept through execution. The candidate is customer obsessed, innovative, independent, results-oriented and enjoys working in a fast-paced growing organization. An interest in operations, manufacturing or process improvement is helpful. The ability to embrace this ambiguity and work with a highly distributed team of experts is critical. While this is a small team, there is opportunity to own globally impactful work and grow your career in technical, programmatic or people leadership. You will likely work with Python or R, though specific particular modelling language. Your problem-solving ability, knowledge of data models and ability to drive results through ambiguity are more important to us. About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. We are open to hiring candidates to work out of one of the following locations: Bangalore, KA, IND
US, WA, Bellevue
The Worldwide Design Engineering (WWDE) organization delivers innovative, effective and efficient engineering solutions that continually improve our customers’ experience. WWDE optimizes designs throughout the entire Amazon value chain providing overall fulfillment solutions from order receipt to last mile delivery. We are seeking a Simulation Scientist to assist in designing and optimizing the fulfillment network concepts and process improvement solutions using discrete event simulations for our World Wide Design Engineering Team. Successful candidates will be visionary technical expert and natural self-starter who have the drive to apply simulation and optimization tools to solve complex flow and buffer challenges during the development of next generation fulfillment solutions. The Simulation Scientist is expected to deep dive into complex problems and drive relentlessly towards innovative solutions working with cross functional teams. Be comfortable interfacing and influencing various functional teams and individuals at all levels of the organization in order to be successful. Lead strategic modelling and simulation projects related to drive process design decisions. Responsibilities: - Lead the design, implementation, and delivery of the simulation data science solutions to perform system of systems discrete event simulations for significantly complex operational processes that have a long-term impact on a product, business, or function using FlexSim, Demo 3D, AnyLogic or any other Discrete Event Simulation (DES) software packages - Lead strategic modeling and simulation research projects to drive process design decisions - Be an exemplary practitioner in simulation science discipline to establish best practices and simplify problems to develop discrete event simulations faster with higher standards - Identify and tackle intrinsically hard process flow simulation problems (e.g., highly complex, ambiguous, undefined, with less existing structure, or having significant business risk or potential for significant impact - Deliver artifacts that set the standard in the organization for excellence, from process flow control algorithm design to validation to implementations to technical documents using simulations - Be a pragmatic problem solver by applying judgment and simulation experience to balance cross-organization trade-offs between competing interests and effectively influence, negotiate, and communicate with internal and external business partners, contractors and vendors for multiple simulation projects - Provide simulation data and measurements that influence the business strategy of an organization. Write effective white papers and artifacts while documenting your approach, simulation outcomes, recommendations, and arguments - Lead and actively participate in reviews of simulation research science solutions. You bring clarity to complexity, probe assumptions, illuminate pitfalls, and foster shared understanding within simulation data science discipline - Pay a significant role in the career development of others, actively mentoring and educating the larger simulation data science community on trends, technologies, and best practices - Use advanced statistical /simulation tools and develop codes (python or another object oriented language) for data analysis , simulation, and developing modeling algorithms - Lead and coordinate simulation efforts between internal teams and outside vendors to develop optimal solutions for the network, including equipment specification, material flow control logic, process design, and site layout - Deliver results according to project schedules and quality Key job responsibilities • You influence the scientific strategy across multiple teams in your business area. You support go/no-go decisions, build consensus, and assist leaders in making trade-offs. You proactively clarify ambiguous problems, scientific deficiencies, and where your team’s solutions may bottleneck innovation for other teams. A day in the life The dat-to-day activities include challenging and problem solving scenario with fun filled environment working with talented and friendly team members. The internal stakeholders are IDEAS team members, WWDE design vertical and Global robotics team members. The team solve problems related to critical Capital decision making related to Material handling equipment and technology design solutions. About the team World Wide Design EngineeringSimulation Team’s mission is to apply advanced simulation tools and techniques to drive process flow design, optimization, and improvement for the Amazon Fulfillment Network. Team develops flow and buffer system simulation, physics simulation, package dynamics simulation and emulation models for various Amazon network facilities, such as Fulfillment Centers (FC), Inbound Cross-Dock (IXD) locations, Sort Centers, Airhubs, Delivery Stations, and Air hubs/Gateways. These intricate simulation models serve as invaluable tools, effectively identifying process flow bottlenecks and optimizing throughput. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Seattle
Amazon's Global Fixed Marketing Campaign Measurement & Optimization (CMO) team is looking for a senior economic expert in causal inference and applied ML to advance the economic measurement, accuracy validation and optimization methodologies of Amazon's global multi-billion dollar fixed marketing spend. This is a thought leadership position to help set the long-term vision, drive methods innovation, and influence cross-org methods alignment. This role is also an expert in modeling and measuring marketing and customer value with proven capacity to innovate, scale measurement, and mentor talent. This candidate will also work closely with senior Fixed Marketing tech, product, finance and business leadership to devise science roadmaps for innovation and simplification, and adoption of insights to influence important resource allocation, fixed marketing spend and prioritization decisions. Excellent communication skills (verbal and written) are required to ensure success of this collaboration. The candidate must be passionate about advancing science for business and customer impact. Key job responsibilities - Advance measurement, accuracy validation, and optimization methodology within Fixed Marketing. - Motivate and drive data generation to size. - Develop novel, innovative and scalable marketing measurement techniques and methodologies. - Enable product and tech development to scale science solutions and approaches. A day in the life - Propose and refine economic and scientific measurement, accuracy validation, and optimization methodology to improve Fixed Marketing models, outputs and business results - Brief global fixed marketing and retails executives about FM measurement and optimization approaches, providing options to address strategic priorities. - Collaborate with and influence the broader scientific methodology community. About the team CMO's vision is to maximizing long-term free cash flow by providing reliable, accurate and useful global fixed marketing measurement and decision support. The team measures and helps optimize the incremental impact of Amazon (Stores, AWS, Devices) fixed marketing investment across TV, Digital, Social, Radio, and many other channels globally. This is a fully self supported team composed of scientists, economists, engineers, and product/program leaders with S-Team visibility. We are open to hiring candidates to work out of one of the following locations: Irvine, CA, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
GB, Cambridge
Our team builds generative AI solutions that will produce some of the future’s most influential voices in media and art. We develop cutting-edge technologies with Amazon Studios, the provider of original content for Prime Video, with Amazon Game Studios and Alexa, the ground-breaking service that powers the audio for Echo. Do you want to be part of the team developing the future technology that impacts the customer experience of ground-breaking products? Then come join us and make history. We are looking for a passionate, talented, and inventive Applied Scientist with a background in Machine Learning to help build industry-leading Speech, Language, Audio and Video technology. As an Applied Scientist at Amazon you will work with talented peers to develop novel algorithms and generative AI models to drive the state of the art in audio (and vocal arts) generation. Position Responsibilities: * Participate in the design, development, evaluation, deployment and updating of data-driven models for digital vocal arts applications. * Participate in research activities including the application and evaluation and digital vocal and video arts techniques for novel applications. * Research and implement novel ML and statistical approaches to add value to the business. * Mentor junior engineers and scientists. We are open to hiring candidates to work out of one of the following locations: Cambridge, GBR
US, TX, Austin
The Workforce Solutions Analytics and Tech team is looking for a senior Applied Scientist who is interested in solving challenging optimization problems in the labor scheduling and operations efficiency space. We are actively looking to hire senior scientists to lead one or more of these problem spaces. Successful candidates will have a deep knowledge of Operations Research and Machine Learning methods, experience in applying these methods to large-scale business problems, the ability to map models into production-worthy code in Python or Java, the communication skills necessary to explain complex technical approaches to a variety of stakeholders and customers, and the excitement to take iterative approaches to tackle big research challenges. As a member of our team, you'll work on cutting-edge projects that directly impact over a million Amazon associates. This is a high-impact role with opportunities to designing and improving complex labor planning and cost optimization models. The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail and outstanding ability in balancing technical leadership with strong business judgment to make the right decisions about model and method choices. Successful candidates must thrive in fast-paced environments, which encourage collaborative and creative problem solving, be able to measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs. Key job responsibilities • Candidates will be responsible for developing solutions to better manage and optimize flexible labor capacity. The successful candidate should have solid research experience in one or more technical areas of Operations Research or Machine Learning. As a senior scientist, you will also help coach/mentor junior scientists on the team. • In this role, you will be a technical leader in applied science research with significant scope, impact, and high visibility. You will lead science initiatives for strategic optimization and capacity planning. They require superior logical thinkers who are able to quickly approach large ambiguous problems, turn high-level business requirements into mathematical models, identify the right solution approach, and contribute to the software development for production systems. • Invent and design new solutions for scientifically-complex problem areas and identify opportunities for invention in existing or new business initiatives. • Successfully deliver large or critical solutions to complex problems in the support of medium-to-large business goals. • Apply mathematical optimization techniques and algorithms to design optimal or near optimal solution methodologies to be used for labor planning. • Research, prototype, simulate, and experiment with these models and participate in the production level deployment in Python or Java. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Austin, TX, USA | Bellevue, WA, USA | Nashville, TN, USA | Seattle, WA, USA | Tempe, AZ, USA
CA, BC, Vancouver
Do you want to be part of the team developing the future technology that impacts the customer experience of ground-breaking products? Then come join us and make history. We are looking for a passionate, talented, and inventive Applied Scientist with a background in AI, Gen AI, Machine Learning, NLP, to help build LLM solutions for Amazon core shopping. Our team works on a variety of projects, including state of the art generative AI, LLM finetuning, alignment, prompt engineering, benchmarking solutions. Key job responsibilities As a Applied Scientist will be expected to work on state of the art technologies which will result in papers publications, however you will not be only theorizing about the algorithms, but you will also have the opportunity to implement them and see how they behave in the field. As a tech lead, this Applied scientist will also be expected to define the research direction, and influence multiple teams to build solutions that improve Amazon and Alexa customer experience. This is an incredible opportunity to validate your research on one of the most exciting Amazon AI products, where assumptions can be tested against real business scenarios and supported by an abundance of data. We are open to hiring candidates to work out of one of the following locations: Vancouver, BC, CAN
US, WA, Seattle
At Amazon, a large portion of our business is driven by third-party Sellers who set their own prices. The Pricing science team is seeking a Sr. Applied Scientist to use statistical and machine learning techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems, helping Marketplace Sellers offer Customers great prices. This role will be a key member of an Advanced Analytics team supporting Pricing related business challenges based in Seattle, WA. The Sr. Applied Scientist will work closely with other research scientists, machine learning experts, and economists to design and run experiments, research new algorithms, and find new ways to improve Seller Pricing to optimize the Customer experience. The Applied Scientist will partner with technology and product leaders to solve business and technology problems using scientific approaches to build new services that surprise and delight our customers. An Applied Scientist at Amazon applies scientific principles to support significant invention, develops code and are deeply involved in bringing their algorithms to production. They also work on cross-disciplinary efforts with other scientists within Amazon. The key strategic objectives for this role include: - Understanding drivers, impacts, and key influences on Pricing dynamics. - Optimizing Seller Pricing to improve the Customer experience. - Drive actions at scale to provide low prices and increased selection for customers using scientifically-based methods and decision making. - Helping to support production systems that take inputs from multiple models and make decisions in real time. - Automating feedback loops for algorithms in production. - Utilizing Amazon systems and tools to effectively work with terabytes of data. You can also learn more about Amazon science here - https://www.amazon.science/ We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, MD, Virtual Location - Maryland
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. This is a part time position, 29 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: Virtual Location - MD
US, NY, New York
Where will Amazon's growth come from in the next year? What about over the next five? Which product lines are poised to quintuple in size? Are we investing enough in our infrastructure, or too much? How do our customers react to changes in prices, product selection, or delivery times? These are among the most important questions at Amazon today. The Topline Forecasting team in the Supply Chain Optimization Technologies (SCOT) group is looking for innovative, passionate and results-oriented Economists to answer these questions. You will have an opportunity to own the long-run outlook for Amazon’s global consumer business and shape strategic decisions at the highest level. The successful candidate will be able to formalize problem definitions from ambiguous requirements, build econometrics models using Amazon’s world-class data systems, and develop cutting-edge solutions for non-standard problems. Key job responsibilities · Develop new econometric models or improve existing approaches using scalable techniques. · Extract data for analysis and model development from large, complex datasets. · Closely work with engineering teams to build scalable, efficient systems that implement prototypes in production. · Apply economic theory to solve business problems in a fast moving environment. · Distill problem definitions from informal business requirements and communicate technical solutions to senior business leaders. · Drive innovation and best practices in applied research across the Amazon research science community. We are open to hiring candidates to work out of one of the following locations: New York, NY, USA
US, WA, Bellevue
We are seeking a passionate, talented, and inventive individual to join the Applied AI team and help build industry-leading technologies that customers will love. This team offers a unique opportunity to make a significant impact on the customer experience and contribute to the design, architecture, and implementation of a cutting-edge product. The mission of the Applied AI team is to enable organizations within Worldwide Amazon.com Stores to accelerate the adoption of AI technologies across various parts of our business. We are looking for a Senior Applied Scientist to join our Applied AI team to work on LLM-based solutions. We are seeking an experienced Scientist who combines superb technical, research, analytical and leadership capabilities with a demonstrated ability to get the right things done quickly and effectively. This person must be comfortable working with a team of top-notch developers and collaborating with our research teams. We’re looking for someone who innovates, and loves solving hard problems. You will be expected to have an established background in building highly scalable systems and system design, excellent project management skills, great communication skills, and a motivation to achieve results in a fast-paced environment. You should be somebody who enjoys working on complex problems, is customer-centric, and feels strongly about building good software as well as making that software achieve its operational goals. Key job responsibilities You will be responsible for developing and maintaining the systems and tools that enable us to accelerate knowledge operations and work in the intersection of Science and Engineering. A day in the life On our team you will push the boundaries of ML and Generative AI techniques to scale the inputs for hundreds of billions of dollars of annual revenue for our eCommerce business. If you have a passion for AI technologies, a drive to innovate and a desire to make a meaningful impact, we invite you to become a valued member of our team. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA