2019 Q4 AWS Machine Learning Research Awards recipients announced

The AWS Machine Learning Research Awards (MLRA) is pleased to announce the 28 recipients of the 2019 Q4 call-for-proposal cycle.

The AWS Machine Learning Research Awards (MLRA) aims to advance machine learning (ML) by funding innovative research and open-source projects, training students, and providing researchers with access to the latest technology. Since 2017, MLRA has supported over 180 research projects from 73 schools and research institutes in 13 countries, with topics such as ML algorithms, computer vision, natural language processing, medical research, neuroscience, social science, physics, and robotics.

On February 18, 2020, we announced the winners of MLRA’s 2019 Q2/Q3 call-for-proposal cycles. We’re now pleased to announce 28 new recipients of MLRA’s 2019 Q4 call-for-proposal cycle. The MLRA recipients represent 26 universities in six countries. The funded projects aim to develop open-source tools and research that benefit the ML community at large, or create impactful research using AWS ML solutions, such as Amazon SageMaker, AWS AI Services, and Apache MXNet on AWS. The following are the 2019 Q4 award recipients:

Congratulations to all MLRA recipients! We look forward to supporting your research.

Recipient

University

Research title

Anasse Bari

New York University

Predictive Analytics and Artificial Intelligence for Social Good

Andrew Gordon Wilson

New York University

Scalable Numerical Methods and Probabilistic Deep Learning with Applications to AutoML

Bo Li

University of Illinois at Urbana-Champaign

Trustworthy Machine Learning as Services via Robust AutoML and Knowledge Enhanced Logic Inference

Dawn Song

University of California, Berkeley

Protecting the Public Against AI-Generated Fakes

Dimosthenis Karatzas

Universitat Autónoma de Barcelona

Document Visual Question Answer (DocVQA) for Large-Scale Document Collections

Dit-Yan Yeung

Hong Kong University of Science and Technology

Temporally Misaligned Spatiotemporal Sequence Modeling

Lantao Liu

Indiana University Bloomington

Environment-Adaptive Sensing and Modeling using Autonomous Robots

Leonidas Guibas

Stanford University

Learning Canonical Spaces for Object-Centric 3D Perception

Maryam Rahnemoonfar

University of Maryland, Baltimore

Combining Model-Based and Data Driven Approaches to Study Climate Change via Amazon SageMaker

Mi Zhang

Michigan State University

DA-NAS: An AutoML Framework for Joint Data Augmentation and Neural Architecture Search

Michael P. Kelly

Washington University

Web-Based Machine Learning for Surgeon Benchmarking in Pediatric Spine Surgery

Ming Zhao

Arizona State University

Enabling Deep Learning across Edge Devices and Cloud Resources

Nianwen Xue

Brandeis University

AMR2KB: Construct a High-Quality Knowledge by Parsing Meaning Representations

Nicholas Chia

Mayo Clinic

Massively-Scaled Inverse Reinforcement Learning Approach for Reconstructing the Mutational History of Colorectal Cancer

Oswald Lanz

Fondazione Bruno Kessler

Structured Representation Learning for Video Recognition and Question Answering

Pierre Gentine

Columbia University

Learning Fires

Pratik Chaudhari

University of Pennsylvania

Offline and Off-Policy Reinforcement Learning

Pulkit Agrawal

Massachusetts Institute of Technology

Curiosity Baselines for the Reinforcement Learning Community

Quanquan Gu

University of California, Los Angeles

Towards Provably Efficient Deep Reinforcement Learning

Shayok Chakraborty

Florida State University

Active Learning with Imperfect Oracles

Soheil Feizi

University of Maryland, College Park

Explainable Deep Learning: Accuracy, Robustness and Fairness

Spyros Makradakis

University of Nicosia

Clustered Ensemble of Specialist Amazon GluonTS Models for Time Series Forecasting

Xin Jin

Johns Hopkins University

Making Sense of Network Performance for Distributed Machine Learning

Xuan (Sharon) Di

Columbia University

Multi-Autonomous Vehicle Driving Policy Learning for Efficient and Safe Traffic

Yi Yang

University of Technology Sydney

Efficient Video Analysis with Limited Supervision

Yun Raymond Fu

Northeastern University

Generative Feature Transformation for Multi-Viewed Domain Adaptation

Zhangyang (Atlas) Wang

Texas A&M University

Mobile-Captured Wound Image Analysis and Dynamic Modeling for Post-Discharge Monitoring of Surgical Site Infection

Zhi-Li Zhang

University of Minnesota

Universal Graph Embedding Neural Networks for Learning Graph-Structured Data

MLRA is now funded though the Amazon Research Awards (ARA) program. Please see the AWS AI call for proposal for more information.

Research areas

Related content

US, WA, Seattle
The Automated Reasoning Group in the Amazon Neuron team is looking for an Applied Scientist to work on the intersection of Artificial Intelligence and program analysis to raise the code quality bar in our state-of-the-art deep learning compiler stack. This stack is designed to optimize application models across diverse domains, including Large Language and Vision, originating from leading frameworks such as PyTorch and JAX. Your role will involve working closely with our custom-built Machine Learning accelerator, Trainium, which represents the forefront of innovation for advanced ML capabilities, and is the underpinning of Generative AI. In this role as an Applied Scientist, you'll be instrumental in designing, developing, and deploying analyzers for ML compiler stages and compiler IRs. You will architect and implement business-critical tooling, publish research, and mentor a brilliant team of experienced scientists and engineers. You will need to be technically capable, credible, and curious in your own right as a trusted AWS Neuron engineer, innovating on behalf of our customers. Your responsibilities will involve tackling crucial challenges alongside a talented engineering team, contributing to leading-edge design and research in compiler technology and deep-learning systems software. Strong experience in programming languages, compilers, program analyzers, theorem provers, and program synthesis engines will be a benefit in this role. A background in machine learning and AI accelerators is preferred but not required.
US, CA, San Francisco
Join the next revolution in robotics at Amazon's Frontier AI & Robotics team, where you'll work alongside world-renowned AI pioneers to push the boundaries of what's possible in robotic intelligence. As a Member of Technical Staff, you'll be at the forefront of developing breakthrough foundation models that enable robots to perceive, understand, and interact with the world in unprecedented ways. You'll drive independent research initiatives in areas such as perception, manipulation, science understanding, locomotion, manipulation, sim2real transfer, multi-modal foundation models and multi-task robot learning, designing novel frameworks that bridge the gap between state-of-the-art research and real-world deployment at Amazon scale. In this role, you'll balance innovative technical exploration with practical implementation, collaborating with platform teams to ensure your models and algorithms perform robustly in dynamic real-world environments. You'll have access to Amazon's vast computational resources, enabling you to tackle ambitious problems in areas like very large multi-modal robotic foundation models and efficient, promptable model architectures that can scale across diverse robotic applications. Key job responsibilities - Drive independent research initiatives across the robotics stack, including robotics foundation models, focusing on breakthrough approaches in perception, and manipulation, for example open-vocabulary panoptic scene understanding, scaling up multi-modal LLMs, sim2real/real2sim techniques, end-to-end vision-language-action models, efficient model inference, video tokenization - Design and implement novel deep learning architectures that push the boundaries of what robots can understand and accomplish - Lead full-stack robotics projects from conceptualization through deployment, taking a system-level approach that integrates hardware considerations with algorithmic development, ensuring robust performance in production environments - Collaborate with platform and hardware teams to ensure seamless integration across the entire robotics stack, optimizing and scaling models for real-world applications - Contribute to the team's technical strategy and help shape our approach to next-generation robotics challenges A day in the life - Design and implement novel foundation model architectures and innovative systems and algorithms, leveraging our extensive infrastructure to prototype and evaluate at scale - Collaborate with our world-class research team to solve complex technical challenges - Lead technical initiatives from conception to deployment, working closely with robotics engineers to integrate your solutions into production systems - Participate in technical discussions and brainstorming sessions with team leaders and fellow scientists - Leverage our massive compute cluster and extensive robotics infrastructure to rapidly prototype and validate new ideas - Transform theoretical insights into practical solutions that can handle the complexities of real-world robotics applications About the team At Frontier AI & Robotics, we're not just advancing robotics – we're reimagining it from the ground up. Our team is building the future of intelligent robotics through innovative foundation models and end-to-end learned systems. We tackle some of the most challenging problems in AI and robotics, from developing sophisticated perception systems to creating adaptive manipulation strategies that work in complex, real-world scenarios. What sets us apart is our unique combination of ambitious research vision and practical impact. We leverage Amazon's massive computational infrastructure and rich real-world datasets to train and deploy state-of-the-art foundation models. Our work spans the full spectrum of robotics intelligence – from multimodal perception using images, videos, and sensor data, to sophisticated manipulation strategies that can handle diverse real-world scenarios. We're building systems that don't just work in the lab, but scale to meet the demands of Amazon's global operations. Join us if you're excited about pushing the boundaries of what's possible in robotics, working with world-class researchers, and seeing your innovations deployed at unprecedented scale.
US, CA, San Francisco
Join the next revolution in robotics at Amazon's Frontier AI & Robotics team, where you'll work alongside world-renowned AI pioneers to push the boundaries of what's possible in robotic intelligence. As a Member of Technical Staff, you'll be at the forefront of developing breakthrough foundation models that enable robots to perceive, understand, and interact with the world in unprecedented ways. You'll drive independent research initiatives in areas such as perception, manipulation, science understanding, locomotion, manipulation, sim2real transfer, multi-modal foundation models and multi-task robot learning, designing novel frameworks that bridge the gap between state-of-the-art research and real-world deployment at Amazon scale. In this role, you'll balance innovative technical exploration with practical implementation, collaborating with platform teams to ensure your models and algorithms perform robustly in dynamic real-world environments. You'll have access to Amazon's vast computational resources, enabling you to tackle ambitious problems in areas like very large multi-modal robotic foundation models and efficient, promptable model architectures that can scale across diverse robotic applications. Key job responsibilities - Drive independent research initiatives across the robotics stack, including robotics foundation models, focusing on breakthrough approaches in perception, and manipulation, for example open-vocabulary panoptic scene understanding, scaling up multi-modal LLMs, sim2real/real2sim techniques, end-to-end vision-language-action models, efficient model inference, video tokenization - Design and implement novel deep learning architectures that push the boundaries of what robots can understand and accomplish - Lead full-stack robotics projects from conceptualization through deployment, taking a system-level approach that integrates hardware considerations with algorithmic development, ensuring robust performance in production environments - Collaborate with platform and hardware teams to ensure seamless integration across the entire robotics stack, optimizing and scaling models for real-world applications - Contribute to the team's technical strategy and help shape our approach to next-generation robotics challenges A day in the life - Design and implement novel foundation model architectures and innovative systems and algorithms, leveraging our extensive infrastructure to prototype and evaluate at scale - Collaborate with our world-class research team to solve complex technical challenges - Lead technical initiatives from conception to deployment, working closely with robotics engineers to integrate your solutions into production systems - Participate in technical discussions and brainstorming sessions with team leaders and fellow scientists - Leverage our massive compute cluster and extensive robotics infrastructure to rapidly prototype and validate new ideas - Transform theoretical insights into practical solutions that can handle the complexities of real-world robotics applications About the team At Frontier AI & Robotics, we're not just advancing robotics – we're reimagining it from the ground up. Our team is building the future of intelligent robotics through innovative foundation models and end-to-end learned systems. We tackle some of the most challenging problems in AI and robotics, from developing sophisticated perception systems to creating adaptive manipulation strategies that work in complex, real-world scenarios. What sets us apart is our unique combination of ambitious research vision and practical impact. We leverage Amazon's massive computational infrastructure and rich real-world datasets to train and deploy state-of-the-art foundation models. Our work spans the full spectrum of robotics intelligence – from multimodal perception using images, videos, and sensor data, to sophisticated manipulation strategies that can handle diverse real-world scenarios. We're building systems that don't just work in the lab, but scale to meet the demands of Amazon's global operations. Join us if you're excited about pushing the boundaries of what's possible in robotics, working with world-class researchers, and seeing your innovations deployed at unprecedented scale.
CN, 31, Shanghai
Team & Project Overview The NBS Data Central team powers analytics, data science, and AI capabilities for Worldwide Global Selling (WWGS). We build scalable data products, and insight-generation systems that drive seller growth across 10+ marketplaces. Seller Intelligence is a P0 foundation theme at the Global Selling level, formed by merging "One Tagging" and "Good Contact" workstreams. It provides seller identity, segmentation, and contact-reach infrastructure that underpins all downstream seller-facing AI workflows — including intelligent outreach, personalized recommendations, and automated engagement. Scope of Impact Own the science pillar for Seller Intelligence within a cross-functional POD (PM + DE + DS + SDE) Directly impact seller engagement metrics across CN, IN, LATAM, and East-Asia expansion regions Models and data products consumed by 5+ downstream teams (ESM, NSR, MKT, NBS AI Ops, ROC) Influence $100M+ annual seller GMS through improved segmentation and contact optimization Key job responsibilities Design and deliver seller segmentation and propensity models at scale — incorporating GMS, category, growth trajectory, engagement signals, and lifecycle stage. Build contact quality scoring and lifecycle management systems (coverage optimization, dormancy detection, reactivation modeling). Define success metrics, experimentation frameworks (A/B, causal inference), and measurement methodology for seller engagement interventions. Productionize ML models and data products — partner with engineering to deploy seller scores, contact quality indices, and recommendation signals. Explore LLM/GenAI applications: automated insight generation from seller data, contact intent classification, and intelligent report synthesis. Serve as the science representative in bi-weekly NBS theme reviews; present findings and proposals to theme Bar Raisers and leadership. Collaborate with BIE team members to democratize analytical outputs via dashboards and self-serve tools. Contribute to cross-marketplace seller behavior analysis supporting Global Expansion strategy (IN, KR, VN, LATAM). Evaluate, integrate, and iterate on AI systems — assess new AI/ML tools, frameworks, and third-party models for applicability to seller intelligence use cases.
US, CA, Palo Alto
The Amazon Search team creates customer-focused search solutions and technologies. Whenever a customer visits an Amazon site worldwide and types in a query or browses through product categories, Amazon Product Search services go to work. We design, develop, and deploy high-performance distributed search systems that rank a catalog of billions of products for hundreds of millions of shoppers. The Search Relevance team owns the ranking models that decide the order of results on every Amazon search page. In this role, you will design and post-train deep ranking models, including LLM-based rankers and multi-tower deep learning models, that jointly optimize purchase, relevance, and personalization. You will invent modeling and training techniques that push the Pareto frontier across multiple objectives, and take your work end to end from novel research prototype through offline evaluation to production online experimentation. Personalization is a first-class objective on this team. You will build models that reason over each customer's history, durable preferences, and query intent to decide which results best fit that specific customer, rather than optimizing a single population-level ranking. We treat search as an active research frontier and invest heavily in staying at the leading edge of ML. Beyond today's ranking stack, our current explorations include LLM agents that reason and plan across multi-step workflows, tool-augmented foundation models, and new paradigms that combine retrieval, reasoning, and personalization. You will help chart where search goes next, and see your ideas ship to real customers within weeks, not quarters. You will work in a dynamic, entrepreneurial team while leveraging the resources of Amazon.com, one of the world's leading technology companies. Please visit https://www.amazon.science for more information. Key job responsibilities Your responsibilities include but are not limited to: - Design, train, and deploy state-of-the-art ranking models that decide how results are ordered on Amazon search, spanning LLM-based rankers and multi-tower deep learning architectures that jointly model engagement, relevance, and personalization. - Post-train LLMs and ranking models with supervised fine-tuning, reinforcement learning (e.g. GRPO, DPO, RLHF), knowledge distillation, and listwise ranking losses (e.g. LambdaLoss, ListNet, ListMLE). - Compose multiple objectives (engagement, relevance, personalization) into a single ranking through principled multi-objective optimization at inference. - Design large-scale label pipelines, including LLM-as-teacher supervision, that turn customer signals and expert judgment into training and reward signals. - Optimize inference for production ranking models through quantization, quantization-aware training, teacher-student distillation, and serving-stack tuning. - Evaluate proposed solutions through offline benchmarks and online A/B tests, and drive the analysis that decides whether a change ships. - Publish and present your work at internal and external scientific venues in ML, NLP, and IR.
IN, KA, Bengaluru
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like the Kindle family of products, Fire Tablets, Fire TV, Health Wellness, Amazon Echo & Astro products. This is an exciting opportunity to join Amazon in developing state-of-the-art techniques that bring Gen AI on edge for our consumer products. We are looking for exceptional scientists to join our Applied Science team and help develop the next generation of edge models, and optimize them while doing co-designed with custom ML HW based on a revolutionary architecture. Work hard. Have Fun. Make History. Key job responsibilities What will you do? - Quantize, prune, distill, finetune Gen AI models to optimize for edge platforms - Fundamentally understand Amazon’s underlying Neural Edge Engine to invent optimization techniques - Analyze deep learning workloads and provide guidance to map them to Amazon’s Neural Edge Engine - Use first principles of Information Theory, Scientific Computing, Deep Learning Theory, Non Equilibrium Thermodynamics - Train custom Gen AI models that beat SOTA and paves path for developing production models - Collaborate closely with compiler engineers, fellow Applied Scientists, Hardware Architects and product teams to build the best ML-centric solutions for our devices - Publish in open source and present on Amazon's behalf at key ML conferences - NeurIPS, ICLR, MLSys.
US, MA, Boston
Are you excited about applying machine learning and applied mathematics to real-world systems at massive scale? As an Applied Scientist on this newly formed team, you will collaborate closely with scientists and engineers to bring research into production across a broad portfolio of problems — from computer vision perception platforms to building-wide optimization and orchestration. You will frame ambiguous business problems as tractable scientific challenges and implement novel machine learning (ML) systems, first-principles models, embedded systems prototypes, and performance optimizations in both prototype and production environments. This is a ground-floor opportunity to shape the scientific direction of a new organization, where your contributions will directly influence how Amazon's fulfillment network operates and evolves. Key job responsibilities - Design, develop, and deploy ML and scientific solutions spanning classical machine learning, statistical modeling, computer vision, optimization, and physics-informed modeling in production environments. - Rapidly ramp on unfamiliar problem domains, frame ambiguous business problems as tractable scientific challenges, and prototype solutions end to end. - Author or co-author research findings for internal or external peer-reviewed venues, and provide peer feedback on research procedures and results across teams. - Prototype and evaluate sensing hardware and lightweight, edge-deployable models that run on commodity compute under real-world constraints. - Collaborate across multiple science and engineering teams to integrate your solutions into deployment architecture, mentoring less experienced scientists along the way. A day in the life You might start your morning reviewing experiment results from an overnight model training run, then shift into a design discussion with engineers on how to deploy a new computer vision model to edge hardware in a fulfillment center. After lunch, you could be prototyping a physics-informed optimization approach, writing up findings for a research paper, or pairing with a teammate to debug a tricky data pipeline. As part of a new and growing organization, you will have a direct hand in shaping team practices, scientific roadmaps, and the tools you use every day. About the team Our team sits within Amazon's fulfillment technology organization and applies a range of scientific disciplines — including computer vision, optimization, reinforcement learning, and statistical modeling — to improve how goods move through Amazon's global fulfillment network. We build the models and systems that drive real-time orchestration, optimizing throughput, flow, and operational performance at scale. As a newly formed organization, we are building our culture and scientific agenda from the ground up. You will join a collaborative, inclusive group of scientists and engineers who value experimentation, rigorous research, and delivering measurable impact for customers.
IN, KA, Bengaluru
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like the Kindle family of products, Fire Tablets, Fire TV, Health Wellness, Amazon Echo & Astro products. This is an exciting opportunity to join Amazon in developing state-of-the-art techniques that bring Gen AI on edge for our consumer products. We are looking for exceptional scientists to join our Applied Science team and help develop the next generation of edge models, and optimize them while doing co-designed with custom ML HW based on a revolutionary architecture. Work hard. Have Fun. Make History. Key job responsibilities What will you do? - Quantize, prune, distill, finetune Gen AI models to optimize for edge platforms - Fundamentally understand Amazon’s underlying Neural Edge Engine to invent optimization techniques - Analyze deep learning workloads and provide guidance to map them to Amazon’s Neural Edge Engine - Use first principles of Information Theory, Scientific Computing, Deep Learning Theory, Non Equilibrium Thermodynamics - Train custom Gen AI models that beat SOTA and paves path for developing production models - Collaborate closely with compiler engineers, fellow Applied Scientists, Hardware Architects and product teams to build the best ML-centric solutions for our devices - Publish in open source and present on Amazon's behalf at key ML conferences - NeurIPS, ICLR, MLSys.
IN, KA, Bengaluru
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like the Kindle family of products, Fire Tablets, Fire TV, Health Wellness, Amazon Echo & Astro products. This is an exciting opportunity to join Amazon in developing state-of-the-art techniques that bring Gen AI on edge for our consumer products. We are looking for exceptional scientists to join our Applied Science team and help develop the next generation of edge models, and optimize them while doing co-designed with custom ML HW based on a revolutionary architecture. Work hard. Have Fun. Make History. Key job responsibilities What will you do? - Quantize, prune, distill, finetune Gen AI models to optimize for edge platforms - Fundamentally understand Amazon’s underlying Neural Edge Engine to invent optimization techniques - Analyze deep learning workloads and provide guidance to map them to Amazon’s Neural Edge Engine - Use first principles of Information Theory, Scientific Computing, Deep Learning Theory, Non Equilibrium Thermodynamics - Train custom Gen AI models that beat SOTA and paves path for developing production models - Collaborate closely with compiler engineers, fellow Applied Scientists, Hardware Architects and product teams to build the best ML-centric solutions for our devices - Publish in open source and present on Amazon's behalf at key ML conferences - NeurIPS, ICLR, MLSys.
GB, London
Come build the future of entertainment with us. Are you interested in shaping the future of movies and television? Do you want to define the next generation of how and what Amazon customers are watching? Prime Video is a premium streaming service that offers customers a vast collection of TV shows and movies — all with the ease of finding what they love to watch in one place. We offer customers thousands of popular movies and TV shows from Originals and Exclusive content to exciting live sports events. We also offer our members the opportunity to subscribe to add-on channels which they can cancel at anytime and to rent or buy new release movies and TV box sets on the Prime Video Store. Prime Video is a fast-paced, growth business — available in over 240 countries and territories worldwide. The team works in a dynamic environment where innovating on behalf of our customers is at the heart of everything we do. If this sounds exciting to you, please read on. Prime Video Commerce's mission is to present the right offer to the right customer at the right time — across subscriptions, channels, and transactional video, in every market and on every device. Our science team replaces static business rules with ML-driven decisions that personalise the entire commerce journey, from discovery through checkout and beyond. We operate at scale across hundreds of millions of customers, and we are expanding into new frontiers — combining the latest advances in agentic and generative AI, behavioural simulation, and causal inference to understand the impact of our decisions before they reach customers. We are looking for an Applied Scientist to join the Prime Video Commerce Insights team in London. You will develop and deploy customer-facing models, understand customer behaviour at scale, and explore emerging techniques that help us make better decisions faster. This is a delivery focused role within a high-visibility multidisciplinary group of engineers and scientists, focused on improving the customer experience for Prime Video. Key job responsibilities - Research, design, and implement machine learning approaches (e.g. reinforcement learning and recommendation systems) that personalise across different customer touch points. - Collaborate with engineers to deploy and integrate successful experiment results into large-scale, complex Amazon production systems with low latency. - Design and execute rigorous experiments to demonstrate the technical efficacy and business value of your methods. - Act as a subject-matter expert and help define the science roadmap and research agenda in line with organisational priorities and production constraints. - Provide machine learning thought leadership to technical and business leaders, thinking strategically about business, product, and technical challenges. - Work with technical product managers to work backwards from what matters to customers and deliver ML-backed solutions. - Share results with the team and wider scientific community through documents that are both statistically rigorous and compellingly relevant. A day in the life You will be a research leader and innovator within the Commerce Insights organisation. You will collaborate with talented engineers and senior leaders to solve problems that are uniquely challenging at Amazon's scale: personalising commerce decisions across multiple business lines, balancing competing objectives, and positively impacting hundreds of millions of customers worldwide. The problems here are technically deep — combining large-scale ML, causal reasoning, and behavioural modelling in a domain where every decision carries real revenue and customer-experience consequences. Your research will ship to production and move metrics that matter. About the team You will join a team of engineers and applied scientists with a proven track record of solving highly complex, ambiguous problems — work that has produced patents and publications at top-tier conferences. The team has direct visibility to senior Prime Video leadership and collaborates broadly across Commerce, Content, and Platform teams to shape how customers discover, subscribe to, and engage with video content. This is a team that operates at the intersection of rigorous research and real-world impact, where your ideas move from whiteboard to production for hundreds of millions of customers.