How to reduce annotation when evaluating AI systems

By exploiting consistencies across components of ensemble classifiers, a new approach reduces data requirements by up to 89%.

Commercial machine learning systems are trained on examples meant to represent the real world. But the world is constantly changing, and deployed machine learning systems need to be regularly reevaluated, to ensure that their performance hasn’t declined.

Evaluating a deployed AI system means manually annotating data the system has classified, to determine whether those classifications are accurate. But annotation is labor intensive, so it is desirable to minimize the number of samples required to assess the system’s performance.

Many commercial machine learning systems are in fact ensembles of binary classifiers; each classifier “votes” on whether an input belongs to a particular class, and the votes are pooled to produce a final decision.

In a paper we’re presenting at the European Conference on Machine Learning, we show how to reduce the number of random samples required to evaluate ensembles of binary classifiers by exploiting overlaps between the sample sets used to evaluate the individual components.

For example, imagine that an ensemble that has three classifiers, and we need 10 samples each to evaluate the performance of the three classifiers. Evaluating the ensemble requires 40 samples — 10 each for the individual classifiers and 10 for the full ensemble. If 10 of the 40 samples were duplicates, we could make do with 30 annotations. Our paper builds on this intuition.

In an experiment using real data, our approach reduced the number of samples required to evaluate an ensemble by more than 89%, while preserving the accuracy of the evaluation.

We also ran experiments using simulated data that varied the degree of overlap between the sample sets for the individual classifiers. In those experiments, the savings averaged 33%.

Finally, in the paper, we show that our sampling procedure doesn’t introduce any biases into the resulting sample sets, relative to random sampling.

Common ground

Intuitively, randomly chosen samples for the separate components of an ensemble would inevitably include some duplicates. Most of the samples useful for evaluating one model should thus be useful for evaluating the others. The goal is to add in just enough additional samples to be able to evaluate all the models.

We begin by choosing a sample set for the entire ensemble, which we dub the “parent”; the individual models of the ensemble are, by reference, “children”. After finding a set of samples sufficient for evaluating the parent, we expand it to include the first child, then repeat the procedure until the set of samples covers all the children.

Our general approach works with any criterion for evaluating an ensemble’s performance, but in the paper, we use precision — or the percentage of true positives that the classifier correctly identifies — as a running example.

Set intersections.png
In this figure, the set of inputs classified as positive by the parent (right circle, AP) intersects the set of inputs classified as positive by the child (left circle, AC). The intersection (orange-shaded region) between a random sample of AP (orange curve, SP) and AC represents S+, the samples from the parent’s positive set that were also classified as positive by the child. The green-shaded region represents S-, samples from the set of inputs that were classified as positive by the child but not the parent. The sprinkled x’s represent Sremain, additional samples of the inputs classified as positive by the child, required to provide enough samples to get a highly accurate estimate of precision.

We begin with the total set of inputs that the parent has judged to belong to the target class and the total set of inputs that the child has. There’s usually considerable overlap between the two sets; for example, in a majority-vote ensemble composed of three classifiers, the ensemble (parent) classifies an input as positive as long as two of the components (children) do.

From the parent set, we select enough random samples to evaluate the parent. Then we find the intersection between that sample set and the child’s total set of positive classifications (S+ in the figure above). This becomes our baseline sample set for the child.

Next, we draw a random sampling of inputs that the child classified as positive but the parent did not (S-, above). The ratio between the size of this sample and the size of the baseline sample set should be the same as the ratio between the number of inputs that the child — but not the parent — labeled positive and the number of inputs that both labeled positive.

When we add these samples to the baseline sample set, we get a combined sample set that may not be large enough to accurately estimate precision. If needed, we select more samples from the inputs classified as positive by the child. These samples may also have been classified as positive by the parent (Sremain in the figure above).

Recall that we first selected samples from the set where the child and parent agreed, then from the set where the child and parent disagreed. That means that the sample set we have constructed is not truly random, so the next step is to mix together the samples in the combined set.

Reshuffle or resample?

We experimented with two different ways of performing this mixing. In one, we simply reshuffle all the samples in the combined set. In the other, we randomly draw samples from the combined set and add them to a new mixed set, until the mixed set is the same size as the combined set. In both approaches, the end result is that when we pick any element from the sample, we won’t know whether it came from the set where the parent and child agreed or the one where they disagreed.

Savings:overlap.png
A visualization of the average savings in samples provided by our approach as we varied the amount of overlap between the parent’s and child’s judgments.

In our experiments, we identified a slight trade-off between the results of our algorithm when we used reshuffling to produce the mixed sample set and when we used resampling. Because resampling introduces some redundancies into the mixed set, it requires fewer samples than reshuffling, which increases the savings in sample size versus random sampling.

At the same time, however, it slightly lowers the accuracy of the precision estimate. With reshuffling, our algorithm, on average, slightly outperformed random sampling on our three test data sets, while with resampling, it was slightly less accurate than random sampling.

Overall, the sampling procedure we have developed reduces the sample size. Of course, the amount of savings depends on the overlap between the parent’s and child’s judgments. The greater the overlap, the greater the savings in samples.

About the Author
Srinivasan Jagannathan is a senior manager of software development at Amazon.

Recommended for you

Work with us

See More Jobs
US, NY, New York
Amazon Web Services is looking for experienced Applied Science Managers to join the Security Analytics and AI ResearchTeam within AWS Security Services. You would be entrusted with researching and developing state-of-the-art data mining and machine learning algorithms for various AWS security services, such as GuardDuty (https://aws.amazon.com/guardduty/) and Macie (https://aws.amazon.com/macie/). On this team, you will help guide and lead team of scientists who invent and implement innovative solutions for never-before-solved problems. This is not solely a management role, you will be directly contributing to the research and development while you lead. If you have passion for security and experience with managing scientists, this will be an exciting opportunity.The AWS Security Services team builds technologies that help customers strengthen their security posture and better meet security requirements in the AWS Cloud. The team interacts with security researchers to codify our own learnings and best practices and make them available for customers. We are building massively scalable and globally distributed security systems to power next generation services.Key Responsibilities:· Lead scientists and engineers on the team and oversee research and development projects at various stages ranging from initial exploration to deployment into production systems.· Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgment.· Collaborate with software engineering teams to integrate successful experiments into large scale, highly complex production services.· Report results in a scientifically rigorous way.· Interact with security engineers, product managers and related domain experts to dive deep into the types of challenges that we need innovative solutions for.
US, WA, Seattle
As a Sr. Research Scientist in Music, you'll discover insights that drive marketing, content and product strategy for our Amazon Music streaming service. You will have the opportunity to work with a huge and rapidly growing data set generated by mobile, desktop and home music streaming application usage interactions and music consumption behaviors.Major responsibilities include:Translate / Interpret· Complex and interrelated datasets describing customer behavior, messaging, content, product design and financial impact.Measure / Quantify / Expand· Retrieve, synthesize, and present critical data in a format that is immediately useful to answering specific questions or improving system performance.· Analyze historical data to identify trends and support decision making.· Improve upon existing methodologies by developing new data sources, testing model enhancements, and fine-tuning model parameters.· Provide requirements to develop analytic capabilities, platforms, and pipelines.· Apply statistical or machine learning knowledge to specific business problems and data.Explore / Enlighten· Formalize assumptions about how users are expected to behave, create statistical definition of the outlier, and develop methods to systematically identify these outliers. Work out why such examples are outliers and define if any actions needed.· Given anecdotes about anomalies or generate automatic scripts to define anomalies, deep dive to explain why they happen, and identify fixes.· Make Decisions and Recommendations· Build decision-making models and propose solution for the business problem you defined· Conduct written and verbal presentation to share insights and recommendations to audiences of varying levels of technical sophistication.· Utilize code (Python/R/SQL) for data analyzing and modeling algorithmsAmazon MusicImagine being a part of an agile team where your ideas have the potential to reach millions. Picture working on cutting-edge consumer-facing products, where every single team member is a critical voice in the decision-making process. Envision being able to leverage the resources of a Fortune-500 company within the atmosphere of a start-up. Welcome to Amazon Music, where ideas are born and come to life as Amazon Music Unlimited, Prime Music, and so much more.Come innovate with the Amazon Music team!
US, WA, Seattle
Are you passionate about conducting research to drive real behavioral change and inform front line managers and leaders in making more effective decisions? Would you love to see your research in practice, impacting Amazonians globally and improving the employee experience? If so, you should consider joining the Amazon Web Services (AWS) Organizational Effectiveness and Science team.AWS is a dynamic, growing business unit within Amazon.com. Since early 2006, AWS has provided companies of all sizes with an infrastructure platform in the cloud. Today, AWS provides a highly reliable, scalable, low-cost infrastructure platform in the cloud that powers millions of active monthly customers from start-ups to multi-national corporations to governments.In this role, you will lead the research strategy to evaluate, diagnose, understand, and surface drivers and moderators for key, global research streams. These include (but are not limited to) productivity, innovation, decision-making, and Amazon culture. You will dive deep to understand what research should be conducted, develop hypotheses that can be tested, and use both quantitative and qualitative research to deliver actionable insights to AWS leaders. For example, a recent research project has looked at how Amazon’s organizational culture spreads throughout the organization by looking at employee-, team-, department-, and country-level characteristics.The ideal candidate must be self-driven, possess a high degree of ownership, have strong business acumen, excellent communication skills, planning/project management skills, and thrive working in a fast-paced environment. This role requires dealing with a high level of ambiguity and feeling comfortable operating in a space with little direction for solving hard, complex problems. Candidates should be comfortable with nested modeling to assess the impact that variables at the team- and organizational-level has on individual-level outcomes. Experience with applying machine learning to develop classification models as well as forecasting models is a plus, but not essential. There is no opportunity in this role to outsource work to others, so candidates must be comfortable taking on a several activities (i.e., identifying data sources, attaining data via SQL, shaping and analyzing the data, and writing up and communicating the results).The successful candidate will also have a demonstrated track record of:· Knowing Your Customers. Understand their priorities, what they are saying, and more importantly, what they aren’t. Deliver and drive value for them.· Innovation. Appreciate what exists, identify why it works. Then rebuild it or build something totally different that derives even MORE value and a better customer experience.· Judgment. Take different perspectives and business needs, develop a solution that works, move the ball forward. Be able to support your opinions with sound reasoning grounded in the business.· Systems Thinker. Understand all the connections and integrations points through the entire talent management lifecycle.· Thinking big. How do we build it bigger, better, faster? What aren’t we thinking of?· Research driven. We seek to be the most scientific HR organization in the world. We form hypotheses about the best talent management techniques and then set out to prove or disprove them with experiments and careful data collection.· Prioritization. There will be a constant flow of work, both tactical and strategic. Determine what gets done first and why, while managing a plan for what to do with everything else.· Building relationships. Partner with AWS HR COE teams, HR Business Partners, Business Leaders, and corporate COE teams. Share best practices, partner on solutions, and move the organizations forward together.· Consulting. Utilize past experience, deep knowledge of processes and policies, and knowing what’s going on across our organization to create custom solutions and localize as needed.· Communicating. Share your ideas, listen to others, follow-up, and follow-up again. Use the language of the business and avoid “consultant speak”.· Resourcefulness. If you don’t know it, that’s ok. But you should know where to go for the answer or how to find out.· Bias for Action and dealing with ambiguity. Sometimes it is not clear how we are going to get there. Can you help carve a path? How fast can you do it? What are the tradeoffs? Take risks and be willing to try new things, fail fast and iterate.KEY RESPONSIBILITIES:· Manage full life cycle of large-scale research programs (i.e., develop strategy, gather requirements, manage, and execute).· Conduct experiments, analytics, and compute behavioral process models for a deep dive understanding of systemic and individual level drivers; combine quantitative and qualitative data to inform research.· Conduct organizational analysis to uncover talent and org effectiveness insights and build talent and organizational effectiveness solutions (both scrappy or long-term) and mechanisms with the objective to help the business increase talent leverage and improve organizational speed and execution.· Partner closely with our Analytics, Employee Survey (“Connections”), and Diversity and Inclusion teams to identify opportunities for integration points and address specific talent needs leveraging talent data and themes.· Partner with AWS Analytics team to automate organizational health reporting and innovative tools to uncover organizational insights to drive strategic talent and organizational discussions.· Measure and assess the impact of tools and resources introduced to the organization. Continually evaluate solutions for quality, business impact, scalability and sustainability. Conduct post launch evaluations to understand successes and improvement opportunities while establishing methods for sustained adoption.· Evaluate research initiatives to provide bottom line value, ROI, and incremental improvements over timeIf you have an entrepreneurial spirit, know how to deliver, are deeply technical, highly innovative and long for the opportunity to build pioneering solutions to challenging problems, we want to talk to you.
US, WA, Seattle
Are you passionate about solving problems in the marketing domain using various scientific methodologies?Are you the one who is full of ideas and ready to make an impact?The Customer Behavior Analytics (CBA) organization owns Amazon’s insights pipeline, from data collection to deep analytics. We aspire to be the place where Amazon teams come for answers, a trusted source for data and insights that empower our systems and business leaders to make better decisions. Our outputs shape Amazon product and marketing teams’ decisions and thus how Amazon customers see, use, and value their experience. CBA-XO’s mission is to make Amazon’s marketing the most measurably effective in the world. Our long-term objective is to measure the incremental impact of all Amazon’s marketing investments on consumer perceptions, actions, and sales. This requires measuring Amazon’s marketing comparably and consistently across channels, business teams and countries using a comprehensive approach that integrates all Paid, Owned and Earned marketing activity. As the experts on marketing performance we will lead the Amazon worldwide marketing community by providing critical cross-country insights that can power marketing best practices and tenets globally.This role will be part of the new High Value Message (HVM) Analytics team within CBA, which is focused on defining and quantifying how effective our marketing campaigns are changing customer’s perception towards Amazon. You will be a leader to define the ambiguous space. We are looking for a thought leader and you demonstrate this by delivering solutions, not just by having ideas. You will work with Economists and apply theory and quantitative methods to relate how our marketing campaigns affect these measurement metrics. We encourage you to shape the business strategy with data driven recommendations. A successful candidate has an entrepreneurial spirit and wants to make a big impact on Amazon shoppers. You will develop strong working relationships and thrive in a collaborative team environment. Your role requires the ability to influence a virtual team of contributors and interact with marketing executives.The main responsibilities for this position include:· Apply expertise in causal modeling to develop systems that describe how Amazon’s marketing campaigns impact shopper’s perceptions.· Review and audit modeling processes and results for other scientists, both junior and senior.· Work with marketing leadership to align our measurement plan with business strategy.· Formalizing assumptions about how models are expected to behave, creating definitions of outliers, developing methods to systematically identify these outliers, and explaining why they are reasonable or identifying fixes for them· Identify new opportunities that are suggested by the data insights.· Bring a department-wide perspective into decision making.
US, WA, Seattle
The Central AWS Econ team is dedicated to bringing the most trustworthy evidence-based analysis to the most strategic decisions for AWS leadership.Our studies impact strategic investments, service business model, resource allocation, product priorities and pricing models, go-to-market motions and more.This economist role partners with AWS business leaders across the organization to define and deliver on economic questions that guide their most strategic decisions. The successful candidate will be a problem solver who enjoys diving into data, is excited by difficult modeling challenges and ambiguous starting points, and possesses strong communication skills to effectively interface and collaborate with product, finance, planning and business teams.Specific questions include developing supporting economics for new business model, evaluating the relationship between short and long term growth, mapping and affecting the customer journey through different AWS products and cloud technologies.The Central AWS Econ team is dedicated to answering these (and many more) questions using quantitative, economic and statistical methods.Key Responsibilities:· Frame and conduct economic studies, from question definition to and communicating practical implications to senior leadership· Develop new repeatable data analysis pipelines to be used by non-economists
US, CA, Virtual Location - California
Are you passionate about leveraging your data science skills to make impact at scale? Do you enjoy developing innovative algorithms, optimization and predictive models to generate insights and recommendations that will be used by millions of Amazon selling partners and FBA operation teams to drive customer impact?Over 2 million Sellers in 10 countries list their products for sale on the Amazon Marketplace. To meet our sellers’ needs, our smart and customer-obsessed employees are constantly innovating and building on new ideas. Fulfillment by Amazon (FBA) is an Amazon service for our sellers. FBA Inbound analytics and data science team partners with FBA inbound product management team to optimize supply chain cost and lead time variability by influencing right trade-offs among cost, speed and FBA supply network capacity.We are looking for a motivated Data Scientist to build, optimize and productionize cutting edge machine learning models. A successful candidate will have strong quantitative, data mining, statistical modeling, machine learning skills and is comfortable facilitating ideation and working from concept through to execution. The position will partner with Product Management, Engineering, Supply Chain optimization and Finance teams to enhance short term and long term business use cases that leverage a range of data science methodologies to solve complex problems for the global FBA Inbound network.A qualified candidate must have demonstrated ability to develop and manage medium to large-scale models and methodologies that are statistically grounded but also functional and practical. Must possess strong written and verbal communication skills, proven ability to engage and collaborate with customers to drive improvements. Possess high intellectual curiosity with ability to quickly learn new concepts/frameworks, algorithms and technology.Key responsibilities of FBA Inbound data scientist include the following:· Research machine learning algorithms and implement by tailoring to FBA business problems· Manipulate/mine data from large databases (Redshift, SQL Server) and create automated pipeline for model training data sets· Improve model usability by analyzing customer behavior and by gathering requirements from business owners and other tech teams.· Create and track accuracy and performance of model predictions/recommendations. Retrain models to maximize business impact· Foster culture of continuous engineering improvement through mentoring, feedback, and analysis.· Lead setting up of machine learning infrastructure and processes for team to collaborate and share codeTo help describe some of our challenges, we created a short video about Supply Chain Optimization at Amazon - http://bit.ly/amazon-scotAmazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation / Age
US, WA, Seattle
Amazon’s eCommerce Foundation (eCF) organization provides the core technologies that drive and power the Amazon website and the consumer experience. Millions of customer page views and orders per day are enabled by the systems eCF builds from the ground up. eCF Data enables business analytics and insights, providing data and data curation capabilities to thousands of internal and external customers worldwide.Amazon is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background to help build industry-leading machine learning services. As a scientist in the BDT team, you'll partner with technology and business teams to build services that surprise and delight our customers. You will be working with petabytes of structured and unstructured data to help our customers derive critical insights and solve real-world problems. You'll design and implement cutting-edge distributed ML services from the ground up, design and run experiments, research new algorithms, and find new ways of optimizing the risk, profitability, and customer experience for a wide variety of business segments across Amazon. As part of this group, you will have the chance to work with a large team of thought leaders, engineers, and scientists in the distributed computing, machine learning, and business intelligence fields.Applied science at Amazon is a fast growing field. This is a highly technical role that requires substantial cross-disciplinary interaction with software engineers, product managers, solution architects, business intelligence engineers, and other scientists. Besides theoretical analysis and innovation, you will work closely with software engineers to put your research, designs, and algorithms into practice. You will also work on cross-disciplinary efforts with other scientists and engineers at Amazon to establish scalable, efficient, automated processes for large-scale data analysis, ML model development, and model validation.We’re looking for top scientists capable of using ML, computer science, distributed systems, and other techniques to design, implement, and evangelize state-of-the-art solutions for previously-unsolved problems.
US, VA, Arlington
Global Talent Management (GTM) is a central Human Resource (HR) team responsible for creating and evolving Amazon’s human capital and talent products and processes. The GTM Science team is a growing interdisciplinary team within GTM that develops evidence-based products and services that power the growth and development of Amazon’s talent across all of our businesses and locations around the world.GTM Science exists to propel GTM and Amazon HR toward being the most scientific HR organization on earth. Our mission is to use science to assist and measurably improve every talent decision made at Amazon. We do this by discovering signals in workforce data, integrating those signals and behavioral recommendations into Amazon’s talent products, and helping Amazonians make high-judgement decisions that raise the bar on talent. Our multi-disciplinary approach covers an array of capabilities, including: data engineering, business intelligence and analytics, research and behavioral sciences, data science, and applied sciences such as economics and machine learning.We are looking for a dynamic leader to join our leadership team. Reporting directly to the Director of GTM Science, you will lead a team of researchers and analysts primarily responsible for handling high-priority senior-leadership research requests that require scientific rigor and rapid response. You will be responsible for building mechanisms to scale collaborations across all areas of GTM Science/Product/Tech and to build project-based partnerships with HR Line Analytics and COE teams. Your approach balances scientific rigor and pragmatism, in order to deliver results at the speed of business decision-making. You and your team thrive on quickly framing ambiguous requests into an actionable research plan and reporting your results to the highest levels of leadership, to meaningfully shape Talent processes, policies, and programs in areas such as: Diversity & Inclusion, Flexible Work, Talent Mobility, Talent Evaluation, Talent Retention, Performance Management.
US, CA, Sunnyvale
Want to build the future of music and audio entertainment?Imagine being part of an agile team, where your ideas have the potential to reach millions. Envision working within a startup atmosphere, while being able to leverage the resources of a Fortune-500 company. Picture working on bleeding-edge consumer-facing products, where every team member is a critical voice in the decision-making process. Welcome to Amazon Music’s New Projects team.Our team builds new experiences for Amazon Music listeners. We help our customers discover up-and-coming creators, while also having access to their favorite music and podcasts. We build systems that are distributed around the world, spanning our music apps, web player, and voice-forward experiences on mobile and Amazon Echo devices, powered by Alexa. Amazon Music products support our mission of delivering audio entertainment in new and exciting ways that listeners love.Amazon Music’s New Projects team is looking for founding team members across a variety of functions, including software engineering/development, product, marketing, design, and more. Come make history, as we launch new projects for millions of listeners.
US, MA, Boston
Data Scientist: Elastic Block Store Data Services (AWS)Come and build the future with us as we change the way customers move and transform their EBS data at never before seen scale.EBS customers frequently need to move or transform their underlying data whether its accelerating volume creation from snapshots using Fast Snapshot Restore, increasing their database storage or changing their volume type through Elastic volumes, or encrypting their volumes using AWS managed keys. Our team creates solutions that enable and simplify EBS customer workflows.Building a High-Performing & Inclusive Team CultureYou should be passionate about working with a world-class team that welcomes, celebrates, and leverages a diverse set of backgrounds and skillsets to deliver results. Driving results is your primary responsibility, and doing so in a way that builds on our inclusive culture is key to our long term success.Work/Life BalanceEBS Data Services values work-life balance. On normal days, our entire team is co-located in the Boston office, but we’re also flexible when people occasionally need to work from home. We generally keep core available hours from 10am to 4pm. Some of the team is available earlier and the rest of us work a little later.Energizing and Interesting Technical ProblemsYou will work in partnership with engineers on the team to build and operate large scale systems that move and transform customer volume data and accelerate access to their data. You’ll be working to provide solutions to both internal and external customers and engage deeply with other teams within EBS, S3, EC2, and many other services. It’s humbling and energizing to provide data movement solutions to customers at AWS scale.Mentorship & Career GrowthWe’re committed to the growth and development of every member of EBS Data Services, and that includes our engineers. You will have the opportunity to contribute to the culture and direction of the entire EBS org and deliver initiatives that will improve the life of all of our teams.EBS Data Services is a growth environment - we’re hiring and scaling rapidly to meet the needs of our customers. You’ll have the opportunity to grow your scope of influence naturally as we scale and work on solutions that impact some of Amazon's largest customers.
US, MA, Virtual Location - Massachuset
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun.The Research and Advanced Development team at Amazon Robotics is seeking interns with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it!
US, NY, New York
Amazon Web Services is looking for world class scientists to join the research group within AWS Security Services. This team is entrusted with researching and developing core data mining and machine learning algorithms for various AWS security services like GuardDuty (https://aws.amazon.com/guardduty/) and Macie (https://aws.amazon.com/macie/). On this team, you will invent and implement innovative solutions for never-before-solved problems. If you have a passion for security and experience with large scale machine learning problems, this will be an exciting opportunity.The AWS Security Services team builds technologies that help customers strengthen their security posture and better meet security requirements in the AWS Cloud. The team interacts with security researchers to codify our own learnings and best practices and make them available for customers. We are building massively scalable and globally distributed security systems to power next generation services.Key Responsibilities:· Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgment· Collaborate with software engineering teams to integrate successful experiments into large scale, highly complex production services· Report results in a scientifically rigorous way· Interact with security engineers and related domain experts to dive deep into the types of challenges that we need innovative solutions for
US, WA, Seattle
#FoundItOnAmazon is to be an upper funnel browsing experience designed to show customers an endless stream of inspirational products. It is a companion to Amazon's search experience and is meant to drive habitual interaction on Amazon and lead to serendipitous purchases. It is deliberately diverse, visually engaging, and meant to encourage customers to experience the thrill of the hunt. Our vision is for #FoundItOnAmazon to be the first destination for customers to find inspiration and drive customers to visit Amazon more often. #FoundItOnAmazon provides inspiration that converts to purchases, and collects engagement signals (follow, hearts) that can improve the customer’s experience across Amazon.The team is looking for an Applied Scientist to be based in Seattle area. As a key member of the team, you will provide machine learning expertise that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance the customer experience. You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.An ideal candidate will be an expert in the areas of machine learning and statistics who will have expertise in applying theoretical models in an applied environment. The candidate will be expected to work on numerous aspects of Machine Learning such as feature engineering, predictive modeling, probabilistic modeling, hyper-parameter tuning, scalable inference methods and latent variable models including transfer learning. Challenges will involve dealing with very large data sets and requirements on throughput.Responsibilities include:· Design, implement, test, deploy, and maintain innovative data and machine learning solutions to accelerate our business.· Create experiments and prototype implementations of new learning algorithms and prediction techniques· Collaborate with scientists, engineers, product managers, and stockholders to design and implement software solutions for science problems· Use machine learning best practices to ensure a high standard of quality for all of the team deliverables
ES, B, Barcelona
Are you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on ML methods and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services. Current research includes machine learning forecast, online reinforcement learning, and anomaly detection models, among others.We are looking for a Machine Learning Scientist with a strong academic background, and expertise in either of the following:· Graph Neural Networks (GNNs), Temporal Graph Networks (TGNs), and/or Graph Deep Learning (GDL)· Probabilistic Machine LearningAt Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where people can find and discover anything they want to buy online. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment.Economists at Amazon Delivery Experience will be expected to work directly with senior management to drive innovation in retail supply chain and help solve key business problems in operations. You will apply the frontier of economic thinking on design, pricing, forecasting, program evaluation, and other areas. You will build econometric models, using our world class data systems, and apply economic theory to solve business problems in a fast moving environment. Economists at Amazon will be expected to develop new techniques to process large data sets, address quantitative problems, and contribute to design of automated systems around the company.
CA, BC, Vancouver
Terms of employment: Full time, permanentJob location: 595 Burrard Street, 12th floor Vancouver, BC V7X 1L4Are you interested in the process that generates prices for hundreds of millions of products globally? Would you like to influence those prices to affect the lives of millions of Amazon users? Amazon's Pricing organization needs an experienced economist to join an R&D team that sets the price for products sold by Amazon. This team models, tests, and owns end-to-end business problems directly affecting the profitability of the company and improving value to our customers. The output of this team has a big impact on how Amazon conducts business, and on the bottom line.Pricing economists apply the frontier of economic thinking to pricing, forecasting, program evaluation, and opportunity quantification. You will build elasticity, demand and other economic models using our world class data systems and will apply economic theory to solve business problems in a fast moving environment. Pricing economists at Amazon are expected to develop new techniques to process large data sets, address quantitative problems, and contribute to design of automated systems performing billions of price evaluations per day. Pricing is creating a new R&D arm in this space offering you the opportunity to directly influence new programs from their inception.The successful candidate for this position will 1) rigorously apply state-of-the-art methods in their field to existing predictive, analytic, and modeling efforts already underway (2) identify new and promising strategy improvements through modeling and testing of novel approaches (3) connect with other economists, research scientists, and business leaders throughout Amazon to ensure Pricing innovations benefit and are benefitted by progress across the company.Work responsibilities· Bring innovative ideas grounded in best practices in your field to bear on a variety of problems in Pricing.· Influence Pricing strategy and direction by providing subject matter expertise in modeling and testing of end-to-end systems.· Recommend mechanisms for evaluating, measuring, and validating the strategies and models you develop.· Make recommendations for adjustments to increase strategy and system value based on model refinements.· Identify areas for continuous improvement and research.Salary $113,200 - $189,200 /yr, commensurate with experience
US, CA, San Francisco
Amazon music offerings are available in multiple countries, and our applications support our mission of delivering music to customers in a way that enhances their day-to-day lives. We can be found on platforms such as the Amazon Echo, Kindle Fire, iOS, and Android as well as on a mixture of home and auto streaming platforms.The Amazon Music Catalog team is responsible for persisting, computing, reconciling and vending Music rights and metadata to other teams and services across Amazon. We process hundreds of millions of updates per day and our services serve tens of thousands of requests per secs to all Amazon Music Customers. We own critical platforms that makes access to the Amazon Music Catalog highly available and accessible to every one.As a research scientist, you will provide machine learning leadership to the team that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance customer experience.You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.Responsibilities:· Drive collaborative research and creative problem solving· Constructively critique peer research and mentor junior scientists and engineers· Create experiments and prototype implementations of new learning algorithms and prediction techniques· Collaborate with engineering teams to design and implement software solutions for science problems· Contribute to progress of the Amazon and broader research communities by producing publications
US, WA, Seattle
Are you excited about driving business growth for millions of sellers through application of Machine Learning and other advanced computer science disciplines? Do you thrive in a fast-moving, large-scale environment that values data-driven decision making and sound scientific practices? We are looking for experienced applied scientists build the next level of intelligence that will help Amazon Marketplace Sellers to succeed and grow their businesses.Amazon Marketplace enables sellers to put their products in front of hundreds of millions of customers and offers sellers the tools and services needed to make e-commerce successful, efficient and simple. Our team is responsible for building the core intelligence, insights, and algorithms that support a broad range of products and features that Amazon Marketplace Sellers depend on. We are tackling large-scale, challenging problems such as helping sellers to prioritize business tasks, and predicting customer demand for new products, by bringing together petabytes of data from diverse sources across Amazon.You should have a proven track-record of delivering solutions using advanced computer science approaches. You will be comfortable using a variety of tools and data sources to answer high-impact business questions that impact millions of customers globally, and be able to break down complex information and insights into clear and concise language and be comfortable presenting your findings to audiences with a broad range of backgrounds.Responsibilities:Develop production software systems utilizing advanced algorithms to solve business problems.Analyze and validate data using data, algorithms, and statistical tools to ensure high data quality and reliable insights.Proactively identify interesting areas for deep dive investigations and future product development.Design and execute experiments, and analyze experimental results in collaboration with Product Managers, Business Analysts, Economists, and other specialists.Partner with data engineering teams across multiple business lines to improve data assets, quality, metrics and insights.Leverage industry best practices to establish repeatable applied science practices, principles & processes.*scijobsAmazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation.
US, NY, New York
Amazon Web Services is looking for world class scientists to join the research team within AWS Security Services. You would be entrusted with researching and developing core data mining and machine learning algorithms for various AWS security services like GuardDuty (https://aws.amazon.com/guardduty/) and Macie (https://aws.amazon.com/macie/). On this team, you will invent and implement innovative solutions for never-before-solved problems. If you have passion for security and experience with large scale machine learning problems, this will be an exciting opportunity.The AWS Security Services team builds technologies that help customers strengthen their security posture and better meet security requirements in the AWS Cloud. The team interacts with security researchers to codify our own learnings and best practices and make them available for customers. We are building massively scalable and globally distributed security systems to power next generation services.Key Responsibilities:· Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgment· Collaborate with software engineering teams to integrate successful experiments into large scale, highly complex production services.· Report results in a scientifically rigorous way· Interact with security engineers and related domain experts to dive deep into the types of challenges that we need innovative solutions for
US, MA, Cambridge
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services, LLCJob Title: Research Scientist ILocation: Cambridge, MAPosition Responsibilities:Participate in the design, development, and evaluation of models and ML (machine learning) technology. Lay the foundation to move from directed interactions to learned behaviors that enable Alexa to proactively take action on behalf of the customer. Interact with various software and business groups to develop an understanding of their business requirements and operational processes. Create computer simulations to support operational decision-making. Identify areas with potential for improvement and work with internal teams to generate requirements to realize improvements. Design optimal or near optimal solution methodologies to be used by in-house decision support tools and software. Create software prototypes to verify and validate the devised solutions methodologies. Integrate prototypes into production systems using standard software development tools and methodologies.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000