MichaelKearnsandAaronRothamazonscholars
Michael Kearns, left, and Aaron Roth, right, are the co-authors ofThe Ethical Algorithm: The Science of Socially Aware Algorithm Design. Kearns and Roth are leading researchers in machine learning, University of Pennsylvania computer science professors, and Amazon Scholars.
University of Pennsylvania

Amazon Scholars Michael Kearns and Aaron Roth discuss the ethics of machine learning

Two of the world’s leading experts on algorithmic bias look back at the events of the past year and reflect on what we’ve learned, what we’re still grappling with, and how far we have to go.

In November of 2019, University of Pennsylvania computer science professors Michael Kearns and Aaron Roth released The Ethical Algorithm: The Science of Socially Aware Algorithm Design. Kearns is the founding director of the Warren Center for Network and Data Sciences, and the faculty founder and former director of Penn Engineering’s Networked and Social Systems Engineering program. Roth is the co-director of Penn’s program in Networked and Social Systems Engineering and co-authored The Algorithmic Foundations of Differential Privacy with Cynthia Dwork. Kearns and Roth are leading researchers in machine learning, focusing on both the design and real-world application of algorithms.

Their book’s central thesis, which involves “the science of designing algorithms that embed social norms such as fairness and privacy into their code,” was already pertinent when the book was released. Fast forward one year, and the book’s themes have taken on even greater significance.

Amazon Science sat down with Kearns and Roth, both of whom recently became Amazon Scholars, to find out whether the events of the past year have influenced their outlook. We talked about what it means to define and pursue fairness, how differential privacy is being applied in the real world and what it can achieve, the challenges faced by regulators, what advice the two University of Pennsylvania professors would give to students studying artificial intelligence and machine learning, and much more.

Q. How has the narrative around designing socially aware algorithms evolved in the past year, and have the events of the past year altered your outlooks in any way?

Aaron Roth: The main thesis of our book, which is that in any particular problem you have to start by thinking carefully about what you want in terms of fairness or privacy or some other social desideratum, and then how you relatively value things like that compared to other things you might care about, such as accuracy—that fundamental thesis hasn't really changed.

Now with the coronavirus pandemic, what we have seen are application areas where how you want to manage the trade-off between accuracy and privacy, for example, is more extreme than we usually see. So, for example, in the midst of an outbreak, contact tracing might be really important, even though you can't really do contact tracing while protecting individual privacy. Because of the urgency of the situation, you might decide to trade off privacy for accuracy. But because the message of our book really is about thinking things through on a case-by-case basis, the thesis itself hasn't changed.

Michael Kearns: The events of the last year, in particular coronavirus, the resulting restrictions on society and the tensions around these restrictions, and all of the recent social upheaval in the United States, clearly has made the topics of our book much more relevant. The book has focused a lot of attention on the use of algorithms for both good and bad purposes, including things like contact tracing or releasing statistics about people's movements or health data, as well as the use of machine learning, AI, and algorithms more generally for applications like surveillance.

Since our book, at a high level, is about the tensions that arise when there's a battle between social norms like equality or privacy and the use of algorithms for optimizing things like performance or error, I don't think anything in the last year has changed our thinking about the technical aspects of these problems. It's clear that society has been forced to face these problems in a very direct way because of the events of the last year, in a way that we really haven't before. In that sense, our timing was very fortunate because the things we're talking about are more relevant now than ever.

Q. How does that affect your ability to define fairness? Is that something that can ever be a fixed definition, or does it need to be adjusted as events or specific use cases dictate?

Kearns: There's not one correct definition of fairness. In every application you have to think about who the parties are that you're trying to protect, and what the harms are that you're trying to protect them from. That changes both over time and in different scenarios.

Roth: Even before the events of the last year, fairness was always a very context- and beholder- dependent notion. One society might be primarily concerned about fairness by race, and another might be primarily concerned about fairness by gender, and a different community might have other norms. The events of the last year have highlighted cases in which not only will things vary over space or communities, but also over time.

People's attitudes about relatively invasive technologies like contact tracing might be quite different now than they were a year ago. If a year ago I told you, “Suppose there was some disease that some people were catching and the most effective way of tamping it down was to do contact tracing.” Many people might have said, “That sounds really invasive to me”, but now that we've all been through one of the alternatives—being on lock down for six months—people's minds might be changed. We’ve definitely seen norms around privacy for health-related data change.

Q. Standard setting bodies have a significant challenge when it comes to auditing algorithms. Given the scope of that challenge, what needs to happen to allow those groups to do that effectively?

Roth: Although it hasn't happened yet, regulatory agencies are thinking about this, and are reaching out to people like us to help them think about doing this in the right way. I don't know of any regulatory agency that is ready yet to audit algorithms at-scale in sensible ways of the technical sort we discuss in the book. But there are regulatory agencies that have gotten the idea that they should be gearing up to do this, and those agencies have started preliminary movements in that direction.

Kearns: Many of the conversations we've had with standard setting bodies make it clear they're realizing that, collectively, they've technologically fallen behind the industries that they regulate. They don't have the right resources or personnel to do some of the more technological types of auditing. But in these conversations, it's also become clear to us that, even if you could snap your fingers and get the right people and the right resources, it will only be part of a broader framework.

Other important pieces involve becoming more precise about best practices, and also thinking carefully about what those specifications should look like. Let me give a concrete example: One of the things that we argue in our book is that there are many laws and regulations in areas like consumer finance, for instance, that try to get at fairness by restricting what kinds of inputs an algorithm can use. These laws and regulations say, “In order to make sure that your model isn't racially discriminatory, you must not use race as a variable.” But, in fact, not using race as a variable is no guarantee that you won't build a model that's discriminatory by race. In fact, it can actually exacerbate that problem. What we advocate in the book is, rather than restricting the inputs, you should specify the behavior you want as outputs. So instead of saying, “Don't use race”, say instead, “The outputs of the models shouldn't be discriminatory by race.”

Q. Differential privacy has progressed from theoretical to applied science in significant ways in the past few years. How is differential privacy being utilized? How does that help balance the trade-off between privacy and accuracy?

Roth: In the last five years or so, differential privacy has gone from an academic topic to a real technology. For example, the 2020 US Decennial Census is going to release all of its statistical products for the first time, subject to the protections of differential privacy. This is because, by law, the Census is required to protect the privacy of the people it is surveying. The ad hoc techniques used in previous decades to protect the statistics have been shown not to work.

I think that what we will see is that the statistics that the Census releases this year will be more protective of the privacy of Americans. However, in the theme of trade off, using rigorous privacy protections is not without cost. Certain kinds of analyses, such as detailed demographic studies that rely on having highly granular Census data, might now be unavailable under differential privacy. We've seen this play out in the public sphere between downstream users of the data and folks at Census who actually have to hammer out the details.

We've seen other interesting uses of differential privacy during the pandemic too. Some tech companies have utilized differential privacy when releasing statistics about personal mobility data gathered during the pandemic. What differential privacy is best at is releasing those kinds of population level statistics: It's exactly designed to prevent you from learning too much about any particular individual. If you want to know how much less people are moving around different cities because of coronavirus restrictions, these data sets let you answer that question without giving up too much privacy for individuals whose mobile devices were providing the data at the most granular level.

Q. So how does differential privacy help protect individual information?

Roth: Oftentimes the things that you will most naturally want to know about a data set are not facts about particular people, but are population level aggregates like, how many people are crowded into my supermarket at 6 a.m. when it opens. If you tell me sufficiently many aggregate statistics, I can do some math and back out particular people's data from that. The fact that aggregate statistics can be disclosive about individual people's data is an unfortunate accident that actually doesn't have too much to do with what you really wanted to learn.

At its most basic level, differential privacy does things like add little bits of noise to the statistics that you're releasing so that what you're telling me is not the exact number of people who were in my local supermarket at 6 a.m., but roughly the number of people who were in the supermarket plus or minus some small number of people. The fortunate mathematical fact is that you can add amounts of noise that are relatively small that still allow you to get good estimates, but are sufficient to wash out the contributions of particular people, making it impossible to learn too much about any particular individual. It lets you get access to these population level questions that you were curious about without incidentally or accidentally learning about particular people, which is the dangerous side.

"We are bullish about algorithms"
Michael Kearns and Aaron Roth talked to Oxford Academic about the future of AI.

Kearns: To make this slightly more concrete, say what I want to do is each day tell everybody how many people were in the supermarket a couple blocks from me at 1 p.m. If you happened to be at that supermarket at one o’clock, then your GPS data is one of the data points that goes into the count. You may consider your presence at supermarket at 1 p.m. to be the kind of private information that you don't want the whole world to know. So then let's say that, on a typical day, there might be a couple hundred people at the supermarket, but that I add a number which is an order of magnitude, plus or minus 25. The addition of that random number mathematically and provably obscures any individual’s contributions to that count. I won't be able to look at that count and try to figure out any particular person who was present. If I add a number that's between minus 25 and 25, I can't affect the overall count by 100. I'll still have an accurate count up to some resolution, but I will have provided privacy to everybody who was present at the supermarket and, actually, all the people who weren't present as well.

Q. How are topics like fairness, accountability, transparency, interpretability, and privacy showing up in computer science curriculum at Penn and elsewhere within higher education?

Kearns: When Aaron and I first started working on the technical aspects of fairness in machine learning and related topics, it was pretty sparsely populated. This was maybe six or seven years ago, and there weren't many papers on the topics. There were some older ones, more from the statistics literature, but there wasn't really a community of any size within machine learning that thought about these problems. On the research side, the opposite is now true. All of the major machine learning conferences have significant numbers of papers and workshops on these topics; they have workshops devoted to these topics. There are now standalone conferences about fairness, accountability, and explainability in machine learning that are growing every year. It's a very vibrant, active research community now. Additionally, even though it's still early, it's an important enough topic that there are now starting to be efforts to teach this even at the undergraduate level.

The last two years at Penn, for example, I have piloted a course called The Science of Data Ethics. It’s deliberately called that and not The Ethics of Data Science. What that represents is that it’s about the science of making algorithms that are more ethical by different norms, like fairness and privacy. It's not your typical engineering ethics course, which at some level is meant to teach you to be a good, responsible person in that you look at case studies where things went wrong and you talk about what you would do differently. This class is a science class. It says: Here are the standard principles of machine learning, here's how those standard principles can lead to discriminatory behavior in my predictive models, and here are alternate principles, or modifications of those principles and the algorithms that implement them, that avoid or mitigate that behavior.

Q. Is there a more multidisciplinary approach to this set of challenges?

Roth: It's definitely a multidisciplinary area. At Penn, we've been actively collaborating with interested folks in the law school and the criminology department. So far, we don't really have interdisciplinary undergraduate courses on these topics. Those courses would be good in the long run, but at the research and graduate level we've been having interdisciplinary conversations for a number of years.

In particular, one critique that we try to anticipate in the book, and that we’re very aware of, is that technical work on making algorithms more ethical is only one piece of a much larger sociological, or what some people would call socio-technical, pipeline.
Michael Kearns

Kearns: Not just at the teaching level, but even in the research community, there's a real melting pot of viewpoints on these topics. Even though our book is focused on the scientific aspects of these issues, we do spend some time mentioning the fact that the science will only take us so far. In particular, one critique that we try to anticipate in the book, and that we’re very aware of, is that technical work on making algorithms more ethical is only one piece of a much larger sociological, or what some people would call socio-technical, pipeline. Machine learning begins with data and ends with a model. But upstream from the data is the entire manner in which the data was collected and the conditions under which it was collected.

One of the things that's very interesting, exciting, and necessary about the dialogue around these kinds of issues is that, even when there's quite a bit to say on them scientifically, you don't want to just put your head down and look at the science. You want to talk to people who are upstream and downstream from the machine learning part of this pipeline because they bring very different perspectives, and can often point out perspectives which can help you change the way you look at things scientifically in a positive way.

Q. If I were a student exploring AI or ML and I wanted to influence this particular conversation, beyond technical skills, what kind of skills should I be developing?

Kearns: What I would very strongly advocate is: think widely, think broadly, think big. Yes, you're going to be doing technical work in particular models and frameworks, and you know you want to get results in those frameworks. But also read what people who are from very, very different fields think about these problems. Go to their conferences, don't just go to the machine learning conferences and to the sub-track on fairness and machine learning. Go to the interdisciplinary conferences and workshops that are deliberately meant to bring together scientists, legal scholars, philosophers, sociologists, and regulators. Hear their views on these problems, keep an ear out for whether they even think you're working on a problem that's relevant or even has a solution.

That's the way I have approached my career: focus on what I'm good at and what I think is interesting from a scientific standpoint, but not in a scientific vacuum. I deliberately expose myself whenever possible to what people from a completely different perspective are thinking about the same set of topics. The good news is that there's a lot of opportunity for that right now. If you work in some branch of material science, it may not be possible to wander out in the world and get diverse perspectives, but everybody has an opinion on AI and machine learning ethics these days, so there is no shortage of sources from which this hypothetical student could go out and find their own technical views challenged or broadened.

Roth: One trap that is very easy for a new PhD student, or even an established researcher, to fall into is to write the introductions to your papers motivated by some kind of fairness problem, but then find yourself solving some narrow technical problem that ultimately has little connection to the world. I am sometimes guilty of this myself, but this is an area where there really are lots of important problems to solve. It's an area where theoretical approaches, if wielded correctly, can be extremely valuable. The thing that’s valuable is to be, sort of, multilingual. It can be difficult to talk to people from other fields because those fields have different vocabularies and a different world view. However, it's important to understand the perspective of these different communities. There are interdisciplinary groups looking at fairness, accountability, and transparency, which bring people together from all sorts of backgrounds to actively work on developing, at the very least, a shared vocabulary—and hopefully a shared world view.

Q. You've become Amazon Scholars fairly recently. What inspired you to take on this role?

Roth: I've spent most of my career as a theorist, so the ways I've been primarily thinking about privacy and fairness are in the abstract. I've had fun thinking about questions like: What kinds of things are, and are not, possible in principle with differential privacy? Or what kinds of semantic fairness promises can you make to people in a way that is still consistent with trying to learn something from the data? The attraction of Amazon and AWS is that it's where the rubber meets the road. Here we are deploying real machine learning products, and the privacy and the fairness concerns are real and pressing.

My hope is that by having a foot in the practice of these problems, not just their theory, not only will I have some effect on how consequential products actually work, but I’ll learn things that will be helpful in developing new theory that is grounded in the real world.

Kearns: I've had a kind of second life in the quantitative finance industry up until I joined Amazon. While I spent time doing practical things in the world of finance, it was more just using my general knowledge in machine learning. The opportunity to come to Amazon and really think about the topics we've been discussing in a practical technological setting seemed like a great opportunity. I'm also a long-term fan and observer of the company. I’ve known people here for many years, and have had great conversations with them. So I’ve watched with great interest over the last decade plus as Amazon grew its machine learning effort from scratch and gradually grew it to have wider and wider applications. It’s now at a point where not only is machine learning used widely within the company to optimize all kinds of processes and recommendations and the like, but it’s also used by customers worldwide in the form of services like Amazon SageMaker.

I have watched this with great interest because when I was studying machine learning in graduate school back in the late 80s, trust me, it was an obscure corner of AI that people kind of raised their eyebrows at. I never would have thought we would reach the point where not only does The Wall Street Journal expect everyone to know what they mean when they write about machine learning, but that it would actually be a product sold at scale.

I've watched these developments from academia and from the world of finance.  It seemed like a great opportunity to combine my very specific current research and other interests with an inside look at one of the great technology companies. Like Aaron, my expectations, which were high, have only been exceeded in the time I've spent here.

Research areas

Amazon Science Newsletter Project Kuiper.jpg
Get more from Amazon Science
Sign up for our monthly newsletter

Work with us

See More Jobs
US, WA, Seattle
Are you passionate about building an experimental culture while creating and delivering differentiated values for our customers? This is a rewarding role where you will be able to draw a clear connection between your work and how it improves customer trust and experience of millions of Amazon customers across globally. Help us reinvent risk management to create a resounding impact for our customers.We are looking for an Applied Scientist to join the Risk Analytics & Insights team and shape our team’s vision, build risk products and execute delivering our big ideas in risk management. In this role, you should be comfortable working independently with the ability to prioritize workloads, remain flexible, and maintain a strong attention to detail in a fast-paced environment. You will evaluate existing and new technologies required for a solution. You will work with a cross-function team (science, product, engineers, risk manager) to apply your specialized expertise to invent, design, and build innovative software that are stable and performant. You will deliver artifacts with the proper level of complexity the first time (or at least minimize incidental complexity). You will make appropriate trade-offs, re-use where possible, and are judicious about introducing dependencies. You are efficient with resource usage (e.g., hardware, data storage, etc.). You will also be involved in the broader scientific community (e.g., give invited talks, lectures, review for conferences).Key role responsibilities include:· Research, design and implement scalable machine learning (ML), natural language, or computational models to solve problems that matter to our customers in an iterative fashion.· Lead the design, implementation, and successful delivery of large, critical, or difficult scientific solutions involving a significant amount of work while delivering solutions with complete independence.· Map business requirements and customer needs to a scientific problem while providing structure around complex risk decision-making problems.· Align the research direction to business requirements and make the right judgments on research and development schedule and prioritization· Contribute to operational excellence in your team’s scientific components, constructively identifying problems and proposing solutions, taking on projects that improve those components, making them better and easier to maintain.· Mentor and develop junior applied scientists/interns or developers who work on data science and ML/AI problems in the same organization.· Stay informed on the latest machine learning, natural language and/or artificial intelligence trends and make presentations to the larger engineering and applied science communities.
DE, BY, Munich
Amazon opened its virtual doors in 1995 and strives to be the world’s most customer-centric company, where customers can find and discover anything they might want to buy online.The Amazon Logistics team is offering positions to creative big thinkers who are passionate about using data to direct decision making and solve complex and large-scale challenges. We are looking for individuals with a strong interest in improving one of the world’s most complex logistic systems. As part of the Amazon Logistics team, we design and build systems that support planning decisions, measure and manage the last mile transportation, real estate investments, and resource planning.As an Operations Research Scientist, you will use your experience to develop new solutions to improve the performance of Amazon Logistics. Working closely with groups such as program managers, operation, business analysts, data scientists and research scientists, you will support various business initiatives and use your experience in modeling, statistics, simulation and optimization to design models of new policies, simulate and optimize their performance, and evaluate their benefits and impacts to cost, reliability, and speed of our outbound transportation network. You will present your findings to your peers and senior management, and work with program managers to integrate the findings into our product plans.This position requires superior analytic thinkers, who are able to quickly approach large ambiguous problems and apply their technical and statistical knowledge to identify opportunities for further research. You should be able to independently mine and analyze data, and be able to use any necessary programming and statistical analysis software to do so. Successful candidates must thrive in fast-paced environments which encourage collaborative and creative problem solving and be able to measure and estimate risks.We’ll expect you to go the extra mile, but we’ll also make sure you’re well rewarded. As well as a competitive salary and stock units, we offer a whole host of other benefits, including an employee discount.There are other, more intangible rewards too. Like our commitment to your development, a refreshing lack of hierarchy, the chance to work with some of the brightest minds in the industry, plenty of team spirit and an informal atmosphere – suits and ties are few and far between.Join our EU team in Luxembourg, London, Munich, Paris, Milan or Madrid.Amazon is an equal opportunities employer. We believe passionately that employing a diverse workforce is central to our success. We make recruiting decisions based on your experience and skills. We value your passion to discover, invent, simplify and build. We welcome applications from all members of society irrespective of age, sex, disability, sexual orientation, race, religion or belief.
GB, London
Amazon opened its virtual doors in 1995 and strives to be the world’s most customer-centric company, where customers can find and discover anything they might want to buy online.The Amazon Logistics team is offering positions to creative big thinkers who are passionate about using data to direct decision making and solve complex and large-scale challenges. We are looking for individuals with a strong interest in improving one of the world’s most complex logistic systems. As part of the Amazon Logistics team, we design and build systems that support planning decisions, measure and manage the last mile transportation, real estate investments, and resource planning.As an Operations Research Scientist, you will use your experience to develop new solutions to improve the performance of Amazon Logistics. Working closely with groups such as program managers, operation, business analysts, data scientists and research scientists, you will support various business initiatives and use your experience in modeling, statistics, simulation and optimization to design models of new policies, simulate and optimize their performance, and evaluate their benefits and impacts to cost, reliability, and speed of our outbound transportation network. You will present your findings to your peers and senior management, and work with program managers to integrate the findings into our product plans.This position requires superior analytic thinkers, who are able to quickly approach large ambiguous problems and apply their technical and statistical knowledge to identify opportunities for further research. You should be able to independently mine and analyze data, and be able to use any necessary programming and statistical analysis software to do so. Successful candidates must thrive in fast-paced environments which encourage collaborative and creative problem solving and be able to measure and estimate risks.We’ll expect you to go the extra mile, but we’ll also make sure you’re well rewarded. As well as a competitive salary and stock units, we offer a whole host of other benefits, including an employee discount.There are other, more intangible rewards too. Like our commitment to your development, a refreshing lack of hierarchy, the chance to work with some of the brightest minds in the industry, plenty of team spirit and an informal atmosphere – suits and ties are few and far between.Join our EU team in Luxembourg, London, Munich, Paris, Milan or Madrid.Amazon is an equal opportunities employer. We believe passionately that employing a diverse workforce is central to our success. We make recruiting decisions based on your experience and skills. We value your passion to discover, invent, simplify and build. We welcome applications from all members of society irrespective of age, sex, disability, sexual orientation, race, religion or belief.
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services LLCTitle: Applied Scientist IIWorksite: Seattle, WAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
The Alexa for Business (A4B) team is building new capabilities into Alexa that help professionals be more productive at work. In addition to teaching Alexa new things, we provide tools that administrators need to manage Alexa at scale across a large organization. These new capabilities are unlocking a whole new set of use cases for Alexa, from the Office to Hotel Rooms, Restaurants and even sports stadiums. Come join us and help us use the power of voice computing to solve completely new challenges!We are looking for passionate data scientists who are excited to solve challenging problems in voice experience modeling as well as help us launch a greenfield initiative in a new domain using the latest ML techniques. As a data scientist on A4B team, you will be developing next-generation voice recognition models products using state of the art machine learning (ML) models with a world class engineering team. You will also work as the leader in experimentation from statistical perspective and heavily influence business decisions.As a member of the A4B organization, you will spend your time as a hands-on machine learning practitioner and an experimentation leader. You will play a key role on the team, building machine learning models from the ground up. At the end of the day, you will have the reward of seeing your contributions benefit millions of A4B customers worldwide.To be successful in this role, you must have a passion for data, efficiency, and accuracy. Specifically, you will· Be the single-threaded leader for all things Data Science for A4B, including (a) shaping the Data Science vision for the business, (b) proposing innovative Think Big ideas that will grow the business and getting them prioritized via planning process, and (c) executing on the Science roadmap.· Build new voice experiences for A4B· Develop and automate metrics and data visualizations that provide insights to the broader organization.· Design and build green field ML models (in partnership with the Product Management and Software Engineering teams) to solve business problems as part of new initiatives in new domains that we are investing in· Be an active contributor in terms of A4B strategic planning, business direction, insight generation, etc.· Regularly share findings and insights, both verbally and written, with the leadership team and in Town Hall settings with the entire org.
US, WA, Seattle
Do you want to join Alexa AI -- the science team behind Amazon’s intelligence voice assistance system? Do you want to utilize cutting-edge deep-learning and machine learning algorithms to delight millions of Alexa users around the world?If your answers to these questions are “yes”, then come join us at the Alexa Artificial Intelligence team, which is in charge of improving Alexa user satisfaction through real-time metrics monitoring and continuous closed-loop learning. The team owns the modules that reduce user perceived defects and frictions through utterance reformulation, contextual and personalized hypothesis ranking.With the Alexa Artificial Intelligence team, you will be working alongside a team of experienced machine/deep learning scientists and engineers to create data driven machine learning models and solutions on tasks such as sequence-to-sequence query reformulation, graph feature embedding, personalized ranking, etc..You will be expected to:· Analyze, understand, and model user-behavior and the user-experience based on large scale data, to detect key factors causing satisfaction and dissatisfaction (SAT/DSAT).· Build and measure novel online & offline metrics for personal digital assistants and user scenarios, on diverse devices and endpoints· Create and innovate deep learning and/or machine learning based algorithms for utterance reformulation and contextual hypothesis ranking to reduce user dissatisfaction in various scenarios;· Perform model/data analysis and monitor user-experienced based metrics through online A/B testing;· Research and implement novel machine learning and deep learning algorithms and models.
US, CA, San Francisco
Can Alexa help anyone experience the music they enjoy? Even if they don't know what they'd like to listen to in this moment? Or, if they know they want “Happy rock from the 90s”, can she help them find it?Your machine learning skills can help make that a reality on the Amazon Music team. We are seeking an Applied Scientist who will join a team of experts in the field of machine learning, and work together to break new ground in the world of understanding and classifying different forms of music, and creating interactive experiences to help users find the music they are in the mood for. We work on machine learning problems for music classification, recommender systems, dialogue systems, NLP, and music information retrieval.You'll work in a collaborative environment where you can pursue ambitious, long-term research, with many peta-bytes of data, work on problems that haven’t been solved before, quickly implement and deploy your algorithmic ideas at scale, understand whether they succeed via statistically relevant experiments across millions of customers, and publish your research. You'll see the work you do directly improve the experience of Amazon Music customers on Alexa/Echo, mobile, and web.The successful candidate will have a PhD in Computer Science with a strong focus on machine learning, or a related field, and 2+ years of practical experience applying ML to solve complex problems in recommender systems, information retrieval, signal processing, NLP or dialogue systems. Great if you have a passion for music, but this is not a requirement.Responsibilities:- Advance long-term, exploratory research projects in machine learning and related fields to create highly innovative customer experiences;- Analyze large amounts of Amazon’s customer data to discover patterns, find opportunities, and develop highly innovative, scalable algorithms to seize these opportunities;- Validate new or improved models via statistically relevant experiments across millions of customers;- Work closely with software engineering teams to build scalable prototypes for testing, and integrate successful models and algorithms in production systems at very large scale.Amazon MusicImagine being a part of an agile team where your ideas have the potential to reach millions. Picture working on cutting-edge consumer-facing products, where every single team member is a critical voice in the decision-making process. Envision being able to leverage the resources of a Fortune-500 company within the atmosphere of a start-up. Welcome to Amazon Music, where ideas are born and come to life as Amazon Music Unlimited, Prime Music, and so much more.Everyone on our team has a meaningful impact on product features, new directions in music streaming, and customer engagement. We are looking for new team members across a variety of job functions including software engineering/development, marketing, design, ops and more. Come join us as we make history by launching exciting new projects in the coming year.Our team is focused on building a personalized, curated, and seamless music experience. We want to help our customers discover up-and-coming artists, while also having access to their favorite established musicians. We build systems that are distributed on a large scale, spanning our music apps, web player, and voice-forward audio engagement on mobile and Amazon Echo devices, powered by Alexa to support our customer base. Amazon Music offerings are available in countries around the world, and our applications support our mission of delivering music to customers in new and exciting ways that enhance their day-to-day lives.Come innovate with the Amazon Music team!
US, WA, Seattle
Organizational Research and Measurement is seeking a multi-disciplinary data scientist to enable and advance our Diversity, Equity and Inclusion work. As Amazon World Wide Consumer increases the speed and complexity of our operations, we need to find new ways to help employees grow and succeed. As a data scientist on the team, you will have the opportunity to work on one of the world’s largest employee data sets and influence the long-term evolution of how we understand and support employees as professionals and individuals.The ideal candidate for this role will be a collaborative team player with strong technical abilities, communication skills, project management experience, and a proactive, start-up-style mentality. You possess solid analytical skills in data science and excels in deriving actionable business insights that drive positive changes. You will have hands-on experience leading product development initiatives, and are able to balance technical leadership with strong business judgment to make the right decisions about technology, models, and methodologies choices. You strive for simplicity, learn quickly, and demonstrate significant creativity and high judgment backed by data.Responsibilities· Enable team data efficacy and efficiency· Lead the data aspects of scientific research projects from start to finish, including experiment / study execution, data gathering and manipulation, synthesis and modeling, reporting, and solution implementation· Design, Plan, build, and maintain experimental and production systems that take inputs from multiple models and data sources to support scalable, iterative, and continuous experimentation· Work with Amazon Operations and HR to identify opportunities for enhancing and enriching data sources to the team· Collaborate with customers, stakeholders, engineers, scientists, and program and product owners· Identify, assess, track, and mitigate data related issues and risks at multiple levels· Seek out and capitalize on opportunities to innovate new data technologies as needs· Identify, apply, and advance best practices across the disciplines of statistics, applied psychology, software development, and data science· Own the advancement of the team’s data analytical capabilities· Identify and pursue collaboration opportunities with other data teams working on parallel or complimentary projects· Manage relationships with other data teams necessary for our team to grow and function· Provide technical leadership in data science and mentoring to team members
US, WA, Seattle
The Organizational Research and Measurement team conducts research supporting all Amazon corporate employees. Our goal is to build talent systems to enable employees to thrive at Amazon. We focus on the entire employee life-cycle to improve both business and employee outcomes. This entails doing longitudinal survey research, organizational network analysis, experimental and quasi-experimental studies for causal inference, building services that plug-in to other tools, and providing general consultation to stakeholders. We aim to improve the outcomes of our business and employees by doing cutting edge social science research.The Role:As a Data Scientist at Amazon, your main focus will be on developing predictive models, simulation, visualization, and support structures for data analysis.
US, WA, Seattle
The Organizational Research and Measurement team conducts research supporting all Amazon corporate employees. Our goal is to build talent systems to enable employees to thrive at Amazon. We focus on the entire employee life-cycle to improve both business and employee outcomes. This entails doing longitudinal survey research, organizational network analysis, experimental and quasi-experimental studies for causal inference, building services that plug-in to other tools, and providing general consultation to stakeholders. We aim to improve the outcomes of our business and employees by doing cutting edge social science research.The Role:As a Data Scientist at Amazon, your main focus will be on developing predictive models, simulation, visualization, and support structures for data analysis.
US, WA, Seattle
The Organizational Research and Measurement team conducts research supporting all Amazon corporate employees. Our goal is to build talent systems to enable employees to thrive at Amazon. We focus on the entire employee life-cycle to improve both business and employee outcomes. This entails doing longitudinal survey research, organizational network analysis, experimental and quasi-experimental studies for causal inference, building services that plug-in to other tools, and providing general consultation to stakeholders. We aim to improve the outcomes of our business and employees by doing cutting edge social science research.The Role:As a Research Scientist at Amazon, you will apply scientific principles, subject matter expertise, and business acumen to deliver results at scale by conducting employee life-cycle research.Key Responsibilities:· Supporting global-scale research initiatives across multiple business segments and implementing a wide range of scientific methodologies to solve stakeholder problems.· Collaborating with a cross-functional team that has expertise in the social sciences (e.g., econometrics, psychometrics, judgement and decision making models), machine learning, data science, data engineering, and business intelligence· Querying from multiple data sources, data cleaning and exploration, and advanced statistical analysis.· Writing high-quality, evidence-based documents that help provide insights to business leaders and gain buy-in.· Convert evidence-based insights into usable products, services, or tools for stakeholders.· Serving as a subject matter expert on a wide variety of topics related to research design, measurement, and analysis.· Sharing knowledge, advocating for innovative solutions, and supporting team members.
US, WA, Seattle
The AWS Central Economics team is looking for a PhD economist. The ideal candidate will have experience with time-series forecasting.You will learn about cloud products, including compute, storage, and databases. You will work on analytic projects requested by senior leadership. You will get the opportunity to learn new techniques. You will be a part of a team with many experienced economists.Amazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation
US, WA, Seattle
The Organizational Research and Measurement (ORM) Team within World Wide Consumer guides the talent strategy for Amazon’s largest workforce comprised of over one million Amazon global employees across order fulfillment, transportation, corporate, consumer, and customer service organizations. We are an interdisciplinary team that combines the talents of science and engineering to develop and deliver data products and solutions across the employee lifecycle including onboarding; high-potential identification; talent development; engagement; movement; retention; and attrition.We are looking for an applied scientist who is able to engineer end-to-end solutions for complex science and business problems around the future of talent evaluation, development, and management at Amazon. You will work closely with I/O psychologists, economists, engineers, and business partners to estimate and validate models on large scale data, and turn the results of these analyses into products, policies, programs, and actions that have a major impact on Amazon’s business. We are looking for creative thinkers who can combine strong data science, software engineering, UX and product development skillsets with a desire to innovate, and who know how to execute and deliver on big ideas.Key Responsibilities· Develop of start-to-finish data product solutions from requirements gathering and ideation, through interface design and implementation.· Design data infrastructure and pipelines for machine learning and analytics products.· Obtain, merge, analyze, and report data using SQL, statistics software, and data visualization tools.· Apply various statistical and machine learning techniques to analyze large and complex data sets.· Communicate applied machine learning and statistic concepts to project sponsors, business leaders, and development teams across Amazon.· Understand business customer needs, iterate on feedback, and drive adoption
US, WA, Seattle
The Organizational Research and Measurement (ORM) Team within World Wide Consumer guides the talent strategy for Amazon’s largest workforce comprised of over one million Amazon global employees across order fulfillment, transportation, corporate, consumer, and customer service organizations. We are an interdisciplinary team that combines the talents of science and engineering to develop and deliver data products and solutions across the employee lifecycle including onboarding; high-potential identification; talent development; engagement; movement; retention; and attrition.We are looking for an applied scientist who is able to engineer end-to-end solutions for complex science and business problems around the future of talent evaluation, development, and management at Amazon. You will work closely with I/O psychologists, economists, engineers, and business partners to estimate and validate models on large scale data, and turn the results of these analyses into products, policies, programs, and actions that have a major impact on Amazon’s business. We are looking for creative thinkers who can combine strong data science, software engineering, UX and product development skillsets with a desire to innovate, and who know how to execute and deliver on big ideas.Key Responsibilities· Develop of start-to-finish data product solutions from requirements gathering and ideation, through interface design and implementation.· Design data infrastructure and pipelines for machine learning and analytics products.· Obtain, merge, analyze, and report data using SQL, statistics software, and data visualization tools.· Apply various statistical and machine learning techniques to analyze large and complex data sets.· Communicate applied machine learning and statistic concepts to project sponsors, business leaders, and development teams across Amazon.· Understand business customer needs, iterate on feedback, and drive adoption
PL, Gdansk
Come and join the Database Migration Accelerator team that helps our customers migrate to the cloud. We are on a mission to transform legacy enterprise workloads into modern AWS native application architectures. We achieve this by utilizing cutting edge tools, sophisticated engineering systems and database expertise. We provide fixed price and high speed migrations to the cloud. Database Migration Accelerator is combining various AWS cloud platform services into one product which would serve our customers.We are a team of professionals that are forward-looking and using latest technology offerings (AWS cloud services, Machine Learning, Mathematical Optimization, Relational and NoSQL databases) to build new capability to operationalize and automate migration methodologies. Databases Services at AWS cover a range of data platforms including: Amazon Aurora, DynamoDB, Redshift, Athena, as well as AWS Database Migration Service, Data Pipeline, Glue and more. As each service grows, so does adoption by customers world-wide.We have an opportunity for a Senior Applied Scientist who is passionate about combining machine learning with developing new offerings for the cloud and is enthusiastic about applying bold new ideas to real-world problems.Joining the AWS Database Services team as a Senior Applied Scientist gives you the opportunity to:· Work for a company that’s at the forefront of the cloud computing space.· Be a part of something unique what no other previously developed and was successful.· Design machine learning solutions to intelligently move enterprises to the cloud.· Truly own the solution from concept design through development to production.· Join the team whose activities are regularly called out publicly by AWS CEO Andy JassyWork/Life BalanceOur team places value on work-life balance. Our team is global, based in the US and Poland. Our Poland teams typically start later in the day to have a couple of hours of overlap with US teams.Mentorship & Career GrowthOur team is dedicated to supporting new team members in an environment that celebrates knowledge sharing and mentorship. Our senior engineers mentor more junior engineers through one-on-one mentoring and collaborative code reviews. Projects and tasks are assigned in a way that leverages your strengths and helps you further develop your skillset.Inclusive Team CultureWe get to build a really cool service and the main contributing factor to our success is the inclusive and welcoming culture that we embody every day.We welcome teammates who are enthusiastic, empathetic, curious, motivated, reliable, and able to collaborate with a diverse team of peers.As a Senior Applied Scientist, your responsibilities will include:· Building new cloud based Machine Learning solutions and algorithms to accelerate migrations to the cloud· Participating in hands-on machine learning experimentation and delivering the results in the form of new products· Creating technical strategies and delivering with limited guidance· Solving difficult and complex software problems. Your solutions should be extensible· Cross-collaborating with a number of different teams with overlapping work, including solutions architects, developers, product managers, senior leaders, and many more· Mentoring more junior members of the team or collaboration partners
PL, Gdansk
Come and join the Database Migration Accelerator team that helps our customers migrate to the cloud. We are on a mission to transform legacy enterprise workloads into modern AWS native application architectures. We achieve this by utilizing cutting edge tools, sophisticated engineering systems and database expertise. We provide fixed price and high speed migrations to the cloud. Database Migration Accelerator is combining various AWS cloud platform services into one product which would serve our customers.We are a team of professionals that are forward-looking and using latest technology offerings (AWS cloud services, Machine Learning, Mathematical Optimization, Relational and NoSQL databases) to build new capability to operationalize and automate migration methodologies. Databases Services at AWS cover a range of data platforms including Amazon Aurora, DynamoDB, Redshift, Athena, as well as AWS Database Migration Service, Data Pipeline, Glue and more. As each service grows, so does adoption by customers world-wide.We have an opportunity for a Senior Applied Scientist who is passionate about mathematical optimization with developing new offering for the cloud and is enthusiastic about applying bold new ideas to real-world problems.Joining the AWS Database Services team as a Senior Applied Scientist gives you the opportunity to:· Work for a company that’s at the forefront of the cloud computing space· Be a part of something unique what no other previously developed and was successful.· Design mathematical optimization algorithms to intelligently move enterprises to the cloud.· Truly own solution from concept design through development to production· Join the team whose activities are regularly called out publicly by AWS CEO Andy JassyWork/Life BalanceOur team places value on work-life balance. Our team is global, based in the US and Poland. Our Poland teams typically start later in the day to have a couple of hours of overlap with US teams.Mentorship & Career GrowthOur team is dedicated to supporting new team members in an environment that celebrates knowledge sharing and mentorship. Our senior engineers mentor more junior engineers through one-on-one mentoring and collaborative code reviews. Projects and tasks are assigned in a way that leverages your strengths and helps you further develop your skillset.Inclusive Team CultureWe get to build a really cool service and the main contributing factor to our success is the inclusive and welcoming culture that we embody every day.We welcome teammates who are enthusiastic, empathetic, curious, motivated, reliable, and able to collaborate with a diverse team of peers.As a Senior Applied Scientist, your responsibilities will include:· Build new cloud based mathematical solutions and algorithms to accelerate migrations to the cloud· Participate in algorithms experimentation and deliver the results in the form of new products· Develop scalable optimization algorithms for moving customer workloads to cloud environments. Create technical strategies and deliver with limited guidance· Creating technical strategies and delivering with limited guidance· Solving difficult and complex software problems. Your solutions should be extensible· Cross-collaborating with a number of different teams with overlapping work, including solutions architects, developers, product managers, senior leaders, and many more· Mentoring more junior members of the team or collaboration partners
US, NY, New York
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services LLCPosition: Data Scientist IILocation: New York, NYPosition Responsibilities:Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Amazon Web Services is looking for world class scientists to join the Security Analytics and AI Research group within AWS Security Services. This team is entrusted with researching and developing core data mining and machine learning algorithms for various AWS security services like GuardDuty (https://aws.amazon.com/guardduty/) and Macie (https://aws.amazon.com/macie/). On this team, you will invent and implement innovative solutions for never-before-solved problems. If you have a passion for security and experience with large scale machine learning problems, this will be an exciting opportunity.The AWS Security Services team builds technologies that help customers strengthen their security posture and better meet security requirements in the AWS Cloud. The team interacts with security researchers to codify our own learnings and best practices and make them available for customers. We are building massively scalable and globally distributed security systems to power next generation services.Key Responsibilities:· Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgment· Collaborate with software engineering teams to integrate successful experiments into large scale, highly complex production services· Report results in a scientifically rigorous way· Interact with security engineers and related domain experts to dive deep into the types of challenges that we need innovative solutions forHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.
AT, Graz
Location: Graz, AustriaDuration: 3-6 monthsAbout us:We are working on the future. If you are seeking an innovative, fast-paced environment where you can apply state-of-the-art technologies to solve extreme-scale real world challenges and provide visible benefit to end-users, this is your opportunity: Come work on the Amazon Prime Air team!We are looking for an outstanding computer vision / machine learning applied scientist who combines superb technical, research and analytical capabilities with a demonstrated ability to get the right things done quickly and effectively. This person must be comfortable working with a team of top-notch applied scientists. We are looking for someone who innovates and loves solving hard problems. You will work hard, have fun, and of course, make history!Export Control License: This position may require a deemed export control license for compliance with applicable laws and regulations. Placement is contingent on Amazon’s ability to apply for and obtain an export control license on your behalf.About those internship roles:Are you inspired by innovation? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? If your answer is yes then you’ll fit right in here. We are a smart team of doers that work passionately to apply cutting edge advances in autonomous drone delivery and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even image yet. We invent new improvements every day. We are Amazon Prime Air and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun.Prime Air at Amazon is seeking a talented and motivated student to join the Prime Air team for an Internship assignment. The candidate will have the opportunity to work with senior engineering staff on existing and new modules and systems. The ideal candidate has solid coding skills, enjoys problem solving and has at least one of the following: strong computer vision skills, strong machine-learning skills, or, strong computer graphics skills.Applicants should have at a minimum one quarter/semester remaining after their internship concludes.As an Applied Scientist intern, you will be responsible for data-driven improvements to our product. Regardless of the team you join, your work will directly impact our customers. You will:· Collaborate with colleagues from science, engineering and business backgrounds.· Present proposals and results in a clear manner backed by data and coupled with actionable conclusions.· Push the state-of-the-art in computer vision, machine learning or computer graphics for large-scale real world problems.· Summarize and present your contributions in a white paper or peer reviewed scientific publication.
US, WA, Seattle
The Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, data scientists and MBAʼs. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement.Roughly 50% of research assistants from previous cohorts have converted to full time data science or economics employment at Amazon. If you are interested, send your CV, transcripts, and a cover letter to our mailing list at econ-internship@amazon.com.Amazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation