Ranking and Calibrating Click-Attributed Purchases in Performance Display Advertising

2017
Download
Abstract
In performance display advertising, bidders compete on behalf of advertisers for ad impressions, that is, the opportunity to display relevant ads on a publisher website. We consider bidding on behalf of online retailers who buy ad impressions hoping to realize value only from purchases attributed from clicks. The bidder has a two stage problem. In the first stage, the bidder has to select a small subset from a large selection of ads, with the selected ads most likely to lead to purchases. In the second stage, the bidder has to estimate the purchase probability of the selected ads, which can then be used to create bid values. The challenge in the first stage is that a model optimized for purchases also needs to be (near) optimal for clicks, due to the click attribution constraint. The challenge in the second stage is that true probability of purchases is extremely small, and is difficult to accurately model. We propose a ranking model, followed by a calibration method, to sequentially address the two stage problem. We describe how ordinal ranking is a natural fit for the ad selection problem and how to learn a single model by optimizing for purchases, while being (near) optimal for clicks. We then propose a calibration method, which comprises of a novel non-uniform binning technique for empirical probability estimation, in conjunction with calibration functions such as isotonic and polynomial regression and Platt scaling. We provide empirical results on logged events from a major ad network, that demonstrate the superiority of ordinal model over binary classifiers for ranking ads and the superiority of our proposed calibration technique over traditional uniform binning based calibration technique.
Research areas
Download

View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Sign up for our monthly newsletter

Work with us

See more jobs
US, WA, Bellevue
Amazon is the 4th most popular site in the US (http://www.alexa.com/topsites/countries/US). Our product search engine is one of the most heavily used services in the world, indexes billions of products, and serves hundreds of millions of customers world-wide. We are working on a new AI-first initiative to re-architect and reinvent the way we do search through the use of extremely large scale next-generation deep learning techniques. Our goal is to make step function improvements in the use of advanced Machine Learning (ML) on very large scale datasets, specifically through the use of aggressive systems engineering and hardware accelerators. This is a rare opportunity to develop cutting edge ML solutions and apply them to a problem of this magnitude. Some exciting questions that we expect to answer over the next few years include:- Can a focus on compilers and custom hardware help us accelerate model training and reduce hardware costs?- Can combining supervised multi-task training with unsupervised training help us to improve model accuracy?- Can we transfer our knowledge of the customer to every language and every locale ? The Search Science team is looking for an Applied Science Manager to drive roadmap on making large business impact through application of Deep Learning models via close collaboration with partner teams. The team also has a focus on data quality measurement, improvement, data bias identification and reduction to achieve model fairness.Success in this role will require the courage to chart a new course. You will manage your own team to understand all aspects of the customer journey. You and your team will inform other scientists and engineers by providing insights and building models to help improving training data quality and reducing bias. The research focus includes but not limited to Natural Language Processing, Search, advertising and more. You will be working with cutting edge technologies that enable big data and parallelizable algorithms. You will play an active role in translating business and functional requirements into concrete deliverables and working closely with software development teams to put solutions into production.
US, WA, Seattle
At Try Before You Buy (TBYB) , our vision is to create a compelling global styling business that becomes customers’ most trusted fashion advisor. In addition to providing the convenience of recurring recommendations through Personal Shopper boxes, we will use customers’ personal preference information to enhance all shopping experiences.Try Before You Buy team at Amazon Fashion is looking for an Applied Scientist to join us to build our next-generation personalized recommendation systems for Personal Shopper and Prime Wardrobe. In this role, you will be responsible for researching, developing, and deploying machine learning, computer vision, and NLP models to make customers' fashion shopping experience at Amazon engaging and joyful.The primary responsibilities of this role include:· Lead complex projects that design and build machine-learning, natural language processing, and computer vision solutions for our customers· Collaborate with PMs in Designers on customer-facing experiences that will utilize this data to transform the apparel shopping experience for Amazon customers.· Perform hands-on data analysis, build machine-learning models, run regular A/B tests, and communicate the impact to senior management.· Drive continued scientific innovation as a thought leader and practitioner.· Provide technical and career development guidance to both scientists and engineers in the organization.· Build ETL pipelines to collect and process data· Frame and transform ambiguous business challenges into science hypotheses. Design and implement offline and online experiments to evaluate them· Develop prototypes to test new concepts/proposals for models and algorithms· Design and build automated, scalable pipelines to train and deploy ML models
US, WA, Seattle
Are you passionate about applying formal verification, program analysis, constraint-solving, and theorem proving to real world problems? Do you want to create products that reduce complexity for customers, increase customer security posture, and are provably correct? If so, then we have an exciting opportunity for you. The EC2 Networking team at Amazon.com is looking for a passionate and innovative Applied Scientist.In this role, you will interact with internal teams and external customers to understand their networking requirements. You will apply your knowledge to propose solutions, create software prototypes, and productize prototypes into production systems using software development tools and methodologies. In addition, you will support and scale your solutions to meet the ever growing demand of customer use.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.This position involves on-call responsibilities, typically for one week every two months. We don’t like getting paged in the middle of the night or on the weekend, so we work to ensure that our systems are fault tolerant. When we do get paged, we work together to resolve the root cause so that we don’t get paged for the same issue twice.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, WA, Seattle
Would you like to shape the future of the video entertainment industry for movies, TV and live sports events? Does solving complex problems within large scale systems excite you? If you answered yes, we have an opportunity for you!Prime Video is disrupting the traditional television and movie industry with a growing library of high-quality media including TV shows, movies and live events including must-see exclusive series like The Boys, Utopia, The Marvelous Mrs. Maisel, The Man in the High Castle, English Premier League, Thursday Night Football and more.To help a growing organization scale features to Prime Video customers, our Forecasting and Capacity Planning organization is innovating on behalf of our global software development team consisting of thousands of engineers. The team is building a predictive scaling solution that will apply machine learning techniques to develop forecasts on key business dimensions and leverage them to inform and automate scaling across our global software environment.You will apply your deep knowledge of data science and feature engineering to concrete problems that have broad cross-organizational, global, and technology impact. Your work will focus on retrieving, cleansing and preparing large scale datasets, training and evaluating models and deploying them to production where we continuously monitor and evaluate. You will work on large engineering efforts that solve significantly complex problems facing global customers. You will be trusted to operate with complete independence and are often assigned to focus on areas where the business and/or architectural strategy has not yet been defined. You must be equally comfortable digging in to business requirements as you are drilling into design with development teams and developing production ready learning models. You consistently bring strong, data-driven business and technical judgment to decisions.You will work with internal and external stakeholders, cross-functional partners, and end-users around the world at all levels. Our team makes a big impact because nothing is more important to us than pleasing our customers, continually earning their trust, and thinking long term. You are empowered to bring new technologies and deep learning algorithms to your solutions. If you crave a sense of ownership, this is the place to be.We embrace the challenges of a fast paced market and evolving technologies, paving the way to universal availability of content and pushing the envelope of streaming video quality. You will be encouraged to see the big picture, be innovative, and positively impact millions of customers. This is a young and evolving business where creativity and drive will have a lasting impact on the way video is enjoyed worldwide.
US, WA, Seattle
We are constantly making Alexa the best voice assistant in the world. Amazon’s Alexa cloud service and Echo devices are used every day, by people you know, in and about their homes. The Alexa Monetization team is hiring talented and experienced Sr. Applied Scientists to help building the next generation products for Alexa across multiple channels and domains. We are seeking an experienced, entrepreneurial, big thinker for a confidential new initiative within Alexa. You will be joining a team doing innovative work, making a direct impact to customers, showing measurable success, and building with the latest natural language processing systems. If you are holding out for an opportunity to:Make a huge impact as an individual· Be part of a team of smart and passionate professionals who will challenge you to grow every day· Solve difficult challenges using your expertise in coding elegant and practical solutions· Create applications at a massive scale used by millions of people· Work with machine learning systems to deliver real experiences, not just researchAnd you are experienced with…· Drive applied science (machine learning) projects end-to-end ~ from ideation, analysis, prototyping, development, metrics, and monitoring· Conduct deep analyses on massive user and contextual data sets· Propose viable modeling ideas to advance optimization or efficiency, with supporting argument, data, or, preferably, preliminary results· Design, develop, and maintain scalable, Machine Learning models with automated training, validation, monitoring and reporting· Stay familiar with the field and apply state-of-the-art Machine Learning techniques to NLP and related optimization problems· Produce peer-reviewed scientific paper in top journals and conferencesAnd you constantly look for opportunities to…· Innovate, simplify, reduce waste, and increase efficiencies· Use data to make decisions and validate assumptions· Automate processes otherwise performed by humans· Learn from others and help grow those around you...then we would love to chat!In 2021, we have the opportunity to build new products and features from the ground up and we are looking for strong, bias for action engineering leaders who are not afraid of taking bold bets and trying new things to improve customer experience for Alexa.As part of a new and growing team, you will be iterating on new features and products to help drive innovation and expansion. You will work on cross-functional and cross-domain opportunities; tackle challenging projects aim to accelerate experimentations in Alexa; and build out operating mechanisms and technology to enable novel customer experiences. You will be instrumental in setting the team culture, quality bar, engineering best practices, and norms. Mentoring and growing the team around you will be one of the primary ways you measure your own success. You will have the opportunity to contribute and develop deep expertise in the areas of distributed systems, machine learning, conversational technologies, user interfaces (including voice and natural user interfaces), data storage and data pipelines.This role is exciting for scientists who love to apply startup mindset to their day-to-day, enjoy working cross-functionally to master both business and technology knowledge, and are passionate about building engineering best practices. If you are looking for opportunity to learn, grow and lead, this is the position for you.
US, CA, Palo Alto
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Amazon's advertising portfolio helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth.This opportunity is for the Advertising Forecasting Science team, which consists of scientists and engineers based in Palo Alto, California. The team builds forecasting models for advertising campaigns and financial planning, with revenue exceeding tens of billions of dollars. In addition, the forecasting science team makes auction prediction and handles bid optimization for billions of requests per day, impacting the company’s performance directly using advanced Machine Learning algorithms.As an Applied Scientist on this team you will:· Drive applied science (machine learning) projects end-to-end - from ideation, analysis, prototyping, development, metrics, and monitoring.· Conduct deep analyses on massive Ad user and contextual data sets· Propose viable modelling ideas to advance optimization or efficiency, with supporting argument, data, or, preferably, preliminary results.· Invent ways to overcome technical limitations and enable new forms of analyses to drive key technical and business decisions.· Design, develop, and maintain scalable, Machine Learning models with automated training, validation, monitoring and reporting.· Stay familiar with the field and apply state-of-the-art Machine Learning techniques to our domain problems, around forecasting, bidding, allocation, and optimization.· Produce peer-reviewed scientific paper in top journals and conferences.· Present results, reports, and data insights to both technical and business leadership.· Work across teams and lead a group of talented scientists and engineers to solve problems in the domains of forecasting, auction theory, bid optimization, and user clustering.Impact and Career Growth:· You will invent new shopper and advertiser experiences, and accelerate the pace of Machine Learning and Optimization.· Influence customer facing shopping experiences to helping suppliers grow their retail business and the auction dynamics that leverage native advertising, this role will be powering the engine of one the fastest growing businesses at Amazon.· Define a long-term science vision for our ad marketplace, driven fundamentally from the needs of our customers, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams.· This is a role that combines science leadership, organizational ability, technical strength, product focus and business understanding.· In the immediate term, this role requires (a) addressing principles of allocation function and pricing in ad marketplace auctions, (b) developing efficient algorithms for multi-objective optimization and AI control methods to find operating points for the ad marketplace auctions and to evolve them, and (c) develop science talent around machine learning, Economics and optimization for WW Advertising.Why you love this opportunity:· Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales.· Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products.· We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.Team video ~ https://youtu.be/zD_6Lzw8raEAmazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us
US, WA, Seattle
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The SP team's primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day!Sponsored Products helps merchants, retail vendors, and brand owners succeed via native advertising that grows incremental sales of their products sold through Amazon. The Sponsored Products Ad Marketplace organization optimizes the systems and ad placements to match advertiser demand with publisher supply using a combination of machine learning, big data analytics, ultra-low latency high-volume engineering systems, and quantitative product focus. Our systems and algorithms operate on one of the world's largest product catalogs, matching shoppers with products - with a high relevance bar and strict latency constraints. Our goals are to help buyers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and to build a major, sustainable business that helps Amazon continuously innovate on behalf of all customers.As a Senior Applied Scientist for the Sponsored Products Detail Page Allocation and Pricing team, you will own systems which make the final decision on which ads to show, where to place them on the page and how many ads to place. This also includes selection of various themes that would appear in detail pages. This is a challenging technical and business problem, which requires us to balance the interests of advertisers, shoppers, and Amazon. You'll develop a data-driven product strategy to define the right quantitative measures of shopper impact, using this to evaluate decisions and opportunities. You'll balance a portfolio of pragmatic and long-term investments to drive long term growth of the ads and retail businesses.As a Senior Applied Scientist on this team you will:· Act as the technical leader in Machine Learning and drive full life-cycle Machine Learning projects.· Develop real-time algorithms to allocate billions of ads per day in advertising auctions.· Lead technical efforts within this team and across other teams.· Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production.· Run A/B experiments, gather data, and perform statistical analysis.· Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving.· Work closely with software engineers to assist in productionizing your ML models.· Research new machine learning approaches.· Recruit Applied Scientists to the team and act as a mentor to other Scientists on the team.Impact and Career Growth:In this role you will have significant impact on this team as well as drive cross team projects that consist of Applied Scientists, Data Scientists, Economists, and Software Development Engineers. This is a highly visible role that will help take our products to the next level. You will work alongside many of the best and brightest science and engineering talent and the work you deliver will have a direct impact on customers and revenue!Why you love this opportunity:Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.Team video ~ https://youtu.be/zD_6Lzw8raE
CA, BC, Vancouver
The Global Specialty Fulfillment (GSF) team that delights millions of customers around the world by delivering our most complex product lines at speeds that redefine the bar. The GSF Strategies Team is a collaborative group of Business Analysts, Business Intelligence Engineers, Data Engineers, Data Scientists, Economists, Product Managers, and Program Managers that creates people-centric innovations this business and their workforce of over a hundred thousand valued associates and leaders.We are now recruiting for an exceptional Economist, Global OperationsThe ideal candidate will be:· A Well-Rounded Athlete – You understand that we succeed or fail as a team. You are always ready to step up beyond your core responsibilities and go the extra mile for the project and your team. You nimbly overcome barriers to deliver the best products more quickly than expected.· A Perpetual Student – You seek knowledge and insight. You turn moments into master’s classes. Whether closing a gap, developing a new skill, or staying ahead of your industry, you revel in the joy of learning and growing.· A Skilled Communicator – You excel when interacting with business and technical partners whether you are chatting, sending a written message, or conducting a presentation.· A Trusted Advisor – You work closely with stakeholders to define key business needs and deliver on commitments. You enable effective decision making by retrieving and aggregating data from multiple sources and compiling it into a digestible and actionable format.· An Inventor at Heart – You innovate on behalf of your customer by proactively implementing improvements, enhancements, and customizations. Your customers marvel at your creative solutions to their unidentified needs.· A Fearless Explorer – You are drawn to take on the hardest problems, navigate ambiguity, and see possibility in what others view with skepticism. You never settle, even in the face of overwhelming obstacles.You will:· Collaborate with business intel and data engineering teams to collect new data and refine of existing data sources to continually improve solutions· Test hypotheses in a high-ambiguity environment making use of qualitative data, judgment, and customer feedback.· Utilize code (Python, R, Scala, etc.) to design, build, and manage scientifically-sound, production-grade models and hands-off-the-wheel solutions that solve specific business problems· Advocate for your customer and align your stakeholders to address our most pressing needs· Distill informal customer requirements into problem definitions· Manage and quantify improvement in customer experience or value for the business resulting from research outcomes· Convey rigorous mathematical concepts and considerations to non-expertsA day in the lifeYou will:· Collaborate with business intel and data engineering teams to collect new data and refine of existing data sources to continually improve solutions· Test hypotheses in a high-ambiguity environment making use of qualitative data, judgment, and customer feedback.· Utilize code (Python, R, Scala, etc.) to design, build, and manage scientifically-sound, production-grade models and hands-off-the-wheel solutions that solve specific business problems· Advocate for your customer and align your stakeholders to address our most pressing needs· Distill informal customer requirements into problem definitions· Manage and quantify improvement in customer experience or value for the business resulting from research outcomes· Convey rigorous mathematical concepts and considerations to non-expertsAbout the hiring groupThe Global Specialty Fulfillment (GSF) team that delights millions of customers around the world by delivering our most complex product lines at speeds that redefine the bar. The GSF Strategies Team is a collaborative group of Business Analysts, Business Intelligence Engineers, Data Engineers, Data Scientists, Economists, Product Managers, and Program Managers that creates people-centric innovations this business and their workforce of over a hundred thousand valued associates and leadersJob responsibilitiesThe Global Specialty Fulfillment (GSF) team that delights millions of customers around the world by delivering our most complex product lines at speeds that redefine the bar. The GSF Strategies Team is a collaborative group of Business Analysts, Business Intelligence Engineers, Data Engineers, Data Scientists, Economists, Product Managers, and Program Managers that creates people-centric innovations this business and their workforce of over a hundred thousand valued associates and leaders.We are now recruiting for an exceptional Economist, Global OperationsThe ideal candidate will be:· A Well-Rounded Athlete – You understand that we succeed or fail as a team. You are always ready to step up beyond your core responsibilities and go the extra mile for the project and your team. You nimbly overcome barriers to deliver the best products more quickly than expected.· A Perpetual Student – You seek knowledge and insight. You turn moments into master’s classes. Whether closing a gap, developing a new skill, or staying ahead of your industry, you revel in the joy of learning and growing.· A Skilled Communicator – You excel when interacting with business and technical partners whether you are chatting, sending a written message, or conducting a presentation.· A Trusted Advisor – You work closely with stakeholders to define key business needs and deliver on commitments. You enable effective decision making by retrieving and aggregating data from multiple sources and compiling it into a digestible and actionable format.· An Inventor at Heart – You innovate on behalf of your customer by proactively implementing improvements, enhancements, and customizations. Your customers marvel at your creative solutions to their unidentified needs.· A Fearless Explorer – You are drawn to take on the hardest problems, navigate ambiguity, and see possibility in what others view with skepticism. You never settle, even in the face of overwhelming obstacles.You will:· Collaborate with business intel and data engineering teams to collect new data and refine of existing data sources to continually improve solutions· Test hypotheses in a high-ambiguity environment making use of qualitative data, judgment, and customer feedback.· Utilize code (Python, R, Scala, etc.) to design, build, and manage scientifically-sound, production-grade models and hands-off-the-wheel solutions that solve specific business problems· Advocate for your customer and align your stakeholders to address our most pressing needs· Distill informal customer requirements into problem definitions· Manage and quantify improvement in customer experience or value for the business resulting from research outcomes· Convey rigorous mathematical concepts and considerations to non-experts
US, WA, Seattle
Prime Video is changing traditional media with an ever-increasing selection of movies, TV shows, Emmy Award winning original content, add-on subscriptions including HBO, and live events like Thursday Night Football. Our architecture operates at exabyte-scale, engineered for reliability, scalability, and performance. Prime Video runs on thousands of device types in over 200 territories worldwide.The Playback Optimization team designs, implements and operates the services and systems responsible for Optimizing and delivering video content to all Prime video customers. Our services and systems are invoked every time our customers click play on the thousands of device types supported by Prime Video in over 240 territories worldwide. We direct gigabits of video data across the Internet every second and ensure that our customers receive the best Optimized playback experience possible. We're building a new and ground-breaking technology from the ground-up to deliver spectacular Optimized video Playback experience to our customers for big live events such as English Premier League and the US Open.As a Senior Data scientist on our team you will wear many hats and work in a highly collaborative environment that’s more startup than big company. We’ll need to tackle complex problems that span a variety of domains: Machine learning, Artificial intelligence, Natural Language processing, real-time and distributed systems and help us build services and systems from ground up which scale and serve billions of requests per day, with obsessively high reliability and low operational overhead. Ideal candidate must demonstrate ability to think strategically and analytically about business, product, and technical challenges to contribute to the development of current and future technical roadmaps for Playback optimization Team. You will help us build the future of advanced analytics across the Prime Video playback, You will partner with Business Intelligence Engineers, Data Engineers, and Software Engineering teams to drive experimentation, forward-looking insights, predictive modeling, and machine learning algorithms. You will build best-in-class analytics solutions at scale. Your work will focus on training and evaluating models and deploying them to production where we continuously monitor and evaluate. You will work on large engineering efforts that solve significantly complex problems facing global customers. You will be trusted to operate with independence and are often assigned to focus on areas with significant impact on audience satisfaction. You must be equally comfortable digging in to customer requirements as you are drilling into design with development teams and developing production ready learning models. You consistently bring strong, data-driven business and technical judgment to decisions. We experiment a lot and it is a must to learn and be curios. You will be encouraged to see the big picture, be innovative, and positively impact millions of customers.You'll work with experienced managers who'll care for you. We'll guide you on your career growth path and there's no shortage of technical challenges.Responsibilities:• Implement code (Python, R, Scala, etc.) for analyzing data and building statistical models to solve specific business problems. Work with software engineering teams to productionize algorithms where appropriate.• Lead the development of the scientific roadmap, guide and develop junior engineers in designing and implementing scientific solutions.• Translate analytic insights into concrete, actionable recommendations for business or product improvement. Develop and present these as reports to senior stakeholders with ranging levels of technical knowledge.• Create, enhance, and maintain technical documentation, and present to other scientists, engineers and business leaders.• Demonstrate thorough technical knowledge on feature engineering of massive datasets, effective exploratory data analysis, and model building to deliver accurate and effective business insights.• Innovate by researching, learning, and adapting new modeling techniques and procedures to existing business problems.• Manage and execute entire project from start to finish including problem solving, data gathering and manipulation, predictive modeling, and stakeholder engagement.
US, MA, Boston
RESEARCH & DESIGN AT AMAZON PHARMACYAt Amazon Pharmacy, we are using design, service, and technology to change the way people think about medicine. Design is a vital part of our company’s DNA — Amazon Pharmacy grew out of PillPack, which has been recognized by Time, Fast Company, and the Cooper Hewitt Museum of Design for our customer experience. At Amazon Pharmacy, you’ll have the opportunity to make a big impact on our customers’ lives while informing the future of pharmacy and contributing to our mission of building the earth’s most customer-centric pharmacy.About the RoleAs a senior researcher, you will own the strategy, roadmap, and delivery of our customer satisfaction metrics (NPS, CSAT), invent new ways of tracking the success of our customer experience over time, and lead quantitative research that provides critical insights for our product and design teams. You will work closely with UX design and writing, product management, engineering, data science, and marketing to deliver programs and solutions that meet our customer and business objectives. This will be a highly visible role managing multiple work streams for our organization. If you obsess over customers, thrive in innovation-centric cultures, and want to be a part of a growing team aspiring to radically improve the experience of health and medication, this role is for you.About YouYou are an organized self-starter that can deal with ambiguity and are always on the lookout for ways to improve the customer experience, invent new approaches within your domain, and scale your impact. You easily move between big picture thinking and obsessing about the details, and have expertise conducting both iterative benchmarking and quantitative deep dives that inform future experiences. You are a confident, creative problem-solver and collaborator that can move from concept to execution with limited direction. You are proactive, results driven, and thrive in a fast-paced, startup-like environment. You are an excellent storyteller and capable of presenting–and defending–your ideas to colleagues, stakeholders, and senior leaders.
US, CA, San Diego
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to help Amazon provide the best customer experience by protecting Amazon customers from hackers and bad actors? Do you want to build advanced algorithmic systems that help manage the trust and safety of millions of customer every day? Are you excited by the prospect of analyzing and modeling terabytes of data and create state-of-art algorithms to solve real world problems? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Amazon Account Integrity team.The Amazon Account Integrity team works to ensure that customers are protected from bad actors trying to access their accounts. Our greatest challenge is protecting customer trust without unjustly harming good customers. To strike the right balance, we invest in mechanisms which allow us to accurately identify and mitigate risk, and to quickly correct and learn from our mistakes. This strategy includes continuously evolving enforcement policies, iterating our Machine Learning risk models, and exercising high‐judgement decision‐making where we cannot apply automation.Main responsibilities· Use statistical and machine learning techniques to create scalable risk management systems· Analyzing and understanding large amounts of Amazon’s historical business data for specific instances of risk or broader risk trends· Design, development and evaluation of highly innovative models for risk management· Working closely with software engineering teams to drive real-time model implementations and new feature creations· Working closely with operations staff to optimize risk management operations· Establishing scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation· Tracking general business activity and providing clear, compelling management reporting on a regular basis· Research and implement novel machine learning and statistical approaches
US, VA, Arlington
Device Economics (DEcon) is looking for an economist experienced in causal inference, machine learning, empirical industrial organization, and scaled systems to advance critical resource allocation and pricing decisions in the Amazon Devices org. This role will help refine and execute scientific vision, including executing tooling to estimate demand. Output will be included in scaled systems and inform Devices finance and business leaders.Amazon Devices designs and builds Amazon first-party consumer electronics products to delight and engage customers. Amazon Devices represents a highly complex space with 100+ products across several product categories (e-readers [Kindle], tablets [Fire Tablets], smart speakers and audio assistants [Echo], wifi routers [eero], and video doorbells and cameras [Ring and Blink]), for sale both online and in offline retailers in several regions. The space becomes more complex with dynamic product offering with new product launches, new marketplace launches, and improvements to existing devices through software improvements. The Devices Economics team leads in analyzing these complex marketplace dynamics to enable science-driven decision making in the Devices org. Device Economics achieves this by combining economic expertise with macroeconomic trends, and including both in scientific applications for use by internal analysts, to provide deep understanding of customer preferences. Our team’s outputs inform product development decisions, investments in future product categories, product pricing and promotion, and bundling across complementary product lines. We have achieved substantial impact on the Devices business, and will achieve more.Amazon devices are durable consumer goods that delight customers at purchase and over time, and exhibit complementarities across product lines and product generations. Device Economics seeks an experienced economist adept in measuring customer preferences and behaviors with proven capacity to innovate, scale measurement, and mentor talent. This role has broad Devices-wide impact by advancing measurement of market dynamics, product interactions, and improving bundling, applicable across all product lines. This role will interact with internal customers.The candidate will work closely with Amazon Devices leadership in finance and product development to refine science roadmaps for innovation and simplification, and advance adoption of insights to influence important resource allocation and prioritization decisions. Excellent communication skills (verbal and written) are required to ensure success of this collaboration. The candidate must be passionate about advancing science for business and customer impact.
US, WA, Seattle
Device Economics (DEcon) is looking for a senior economist with deep experience in empirical industrial organization and program evaluation to lead scientific evaluation and improvement of metrics used in promotion of Amazon Devices. The ideal candidate will be comfortable combining economic theory and empirical analyses to assess existing metrics in a principled manner, and devise superior metrics to advance business and customer interests. This role will be a critical leader in crafting the empirical strategy to advance promotion of first party Amazon Devices within the Device Economics team.Amazon Devices designs and builds Amazon first-party consumer electronics products to delight and engage customers. Amazon Devices represents a highly complex space with 100+ products across several product categories (e-readers [Kindle], tablets [Fire Tablets], smart speakers and audio assistants [Echo], wifi routers [eero], and video doorbells and cameras [Ring and Blink]), for sale both online and in offline retailers in several regions. The space becomes more complex with dynamic product offering with new product launches, new marketplace launches, and improvements to existing devices through software improvements. Importantly, the Amazon Devices business operates as a manufacturer, with unique complexities distinct from the retail business requiring new science and innovation. The space remains ‘Day 1’.The Devices Economics team leads in analyzing the complex marketplace dynamics and advancing science to reflect the durable goods aspect, and the broader availability of products (online and offline promotion). Device Economics enables science-driven decision making by combining economic expertise with macroeconomic trends, and including both in scientific applications for use by internal analysts, to provide deep understanding of customer preferences. Our team’s outputs inform product development decisions, promotion algorithms, investments in future product categories, product pricing and promotion, and bundling across complementary product lines. We have achieved substantial impact on the Devices business, and will achieve more.Device Economics seeks an expert in valuing durable goods to help the business optimize promotion and merchandising of our products to deliver customers. This expert will evaluate existing promotion metrics, propose theoretical framework to combine different metrics, and drive a research agenda to advance the metrics. Applications could include promotion algorithms, product assortment offline, bundling recommendation, and for hypothesis generation of value drivers.The candidate will work closely with Amazon Devices finance and promotion leadership to devise science roadmaps for innovation and simplification, and be comfortable operating in a cross-functional environment. Excellent communication skills (verbal and written) are required to ensure success of this collaboration. The candidate must be passionate about advancing science for business and customer impact.
US, MA, North Reading
The Research and Advanced Development team at Amazon Robotics is seeking passionate, hands-on scientists to work on cutting edge algorithms to power automation in Amazon’s order fulfillment and transportation network. Our multi-disciplinary team includes scientists with backgrounds in AI planning and scheduling, robotic grasping and manipulation, machine learning, and operations research. We work on problems such as:- Dynamic allocation and scheduling of tasks to thousands of robots- Learning how to manipulate all the products that Amazon sells- Planning and coordinating the paths of thousands of robots- Co-design of robotic logistics processes and the algorithms to optimize themThe ideal candidate for this position will be familiar with planning or learning algorithms at both the theoretical and implementation levels. You will have the chance to solve complex scientific problems and see your solutions come to life in Amazon’s warehouses!
US, CA, Sunnyvale
Data Scientist Job DescriptionAre you interested in making history, defining and building innovative products and services for the robotics industry?AWS Robotics is a service that makes it easy for developers to build, test, and deploy intelligent robotics applications at scale. The service provides robotics software framework, development, simulation, and fleet management capabilities as an integrated platform. We are looking for a passionate Data Scientist to guide the development of new machine learning solutions for robotics.Job OverviewAs a Senior Data Scientist, you work closely with other research scientists, machine learning experts, and business experts to design and run experiments, research new algorithms, and find new ways to improve AWS’s robotic solutions. The Scientist will partner with technology and product leaders to solve business and technology problems using scientific approaches to build new services that surprise and delight our customers. Science at AWS is a highly experimental activity, although theoretical analysis and innovation are also welcome. Our scientists work closely with software engineers to put algorithms into practice. They also work on cross-disciplinary efforts with other scientists within AWS. You will have a significant role in developing others.The key strategic objectives for this role include:· Understanding drivers, impacts, and key influences on computer vision systems and robotic automation dynamics.· Drive actions at scale to improve the efficiency, reliability and predictability of robotic automation solutions using scientifically-based methods and decision making.· Helping to build production systems that take inputs from multiple models and make decisions in real time.· Automating feedback loops for algorithms in production.· Utilizing Amazon systems and tools to effectively work with petabytes of data.
US, WA, Virtual Location - Washington
The intern will be placed in a large economic team with 10+ full time economists and 3 other interns in the cohort.A day in the lifeTalk with business people to understand business context.Brainstorm with peer economists.Network with other economists and scientists on the team.About the hiring groupThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.Job responsibilitiesThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, data scientists and MBAʼs. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement.Roughly 50% of research assistants from previous cohorts have converted to full-time employment at Amazon.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, WA, Virtual Location - Washington
The intern will be placed in a large economic team with 10+ full time economists and 3 other interns in the cohort.A day in the lifeTalk with business people to understand business context.Brainstorm with peer economists.Network with other economists and scientists on the team.About the hiring groupThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.Job responsibilitiesThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, data scientists and MBAʼs. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement.Roughly 50% of research assistants from previous cohorts have converted to full time employment at Amazon.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, WA, Virtual Location - Washington
The intern will be placed in a large economic team with 10+ full time economists and 3 other interns in the cohort.A day in the lifeTalk with business people to understand business context.Brainstorm with peer economists.Network with other economists and scientists on the team.About the hiring groupThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.Job responsibilitiesThe Amazon Economics Team is hiring Interns in Economics. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, UNIX, and Sawtooth would be a plus.These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, data scientists and MBAʼs. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement.Roughly 50% of research assistants from previous cohorts have converted to full time data science or economics employment at Amazon.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, CA, San Francisco
About Us:Launched in 2011, Twitch is a global community that comes together each day to create multiplayer entertainment: unique, live, unpredictable experiences created by the interactions of millions. We bring the joy of co-op to everything, from casual gaming to outstanding esports to anime marathons, music, and art streams. Twitch also hosts TwitchCon, where we bring everyone together to celebrate and grow their personal interests and passions. We're always live at Twitch. Keep an eye on all things Twitch on LinkedIn, Twitter and on our Blog.About the Role:Twitch Commerce helps communities grow by allowing viewers to show their monetary support to creators through patronage. We take pride in our mission to help creators build communities and earn a living entertaining them. The scale and diversity of users, transactions, social interactions and communities presents an exciting opportunity to use our data to personalize experiences for our users.The Commerce ML team is chartered to build and operate the ML models underpinning Twitch Commerce. For example, we work with our product teams to create production models that protect our business and users from fraud and invigorate the financial vitality of our communities by making every gift subscription matter.As a Senior Applied Scientist on the Commerce ML team, you will build solutions and experiments at scale. Twitch data is unique-- granular structured event data, graphs of user interactions, raw live video, and chat. Using it to make a difference for our customers is a rewarding technical challenge for the right contributor leading to immediate monetary impact for creators and for Twitch. This is a green-field territory that combines science leadership, technical strength, and product empathy. Come join us!You Will:· Contribute to and own a piece of the long-term machine learning vision for Commerce, motivated by the needs of our customers, translating that direction into specific plans for applied scientists, s engineering, product, and design teams.· Design, prototype and implement Machine Learning (ML) algorithms· Develop the production data pipelines and ML pipelines on which those algorithms operate· Work with investigators, analysts, PMs and SDEs to turn ever-evolving insights about our users into automated services.· Mentor junior scientists and engineers
US, WA, Seattle
If you are you seeking an environment where you can drive innovation, collaborate in a team of +30 researchers, applied and data scientists, move marketing science to the edge of the industry to make sure our customers receive the advertising they vote for: keep reading.We look for inquisitive, life long learner with a keen interest on evidence driven decision making, e-commerce and digital businesses and understanding of customer behaviour. If this is you, read on!We are looking for a Senior Research Scientist that will help us create best measurement and optimization models for our marketing functions on all consumer businesses in Amazon across all geographies. This includes· Identification of stakeholder business question and potential technical solutions;· Designing POC and POC performance metrics;· Road-mapping the project and scoping the work;· Delegating tasks;· Coordinating with stakeholders during POC and productionization process· Authoring working paper and other documentation.· For specified research objective, derive new, or modified, statistical estimators or identify appropriate statistical estimator.· Deriving risk function and identifying data conditions under which proposed estimator is better than other candidate methods.· Identifying/designing checks to ensure empirical data environment meets the data criteria for proposed estimator to be superior to other candidate methods· New potential statistical solutions to identified business goals. Conscious risk taker, opportunity driven motivated by innovative approaches taking you and your team out of your comfort zone.· Establish standards across teams to ensure consistency of model outputs across portfolio of solutions.· Ability to track whole cycle of modeling: from data inputs, modeling techniques, diagnosis on outputs (interpretation and stability) and productionalization· Mentoring - Be an unblockerAmazon Science gives you insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Our scientists continue to publish, teach, and engage with the academic community, in addition to utilizing our working backwards method to enrich the way we live and work.Please visit https://www.amazon.science for more information.Amazon is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation / Age. Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.