On-demand container loading in AWS Lambda

2023
Download Copy BibTeX
Copy BibTeX
AWS Lambda is a serverless event-driven compute service, part of a category of cloud compute offerings sometimes called Function-as-a-service (FaaS). When we first released AWS Lambda, functions were limited to 250MB of code and dependencies, packaged as a simple compressed archive. In 2020, we released support for deploying container images as large as 10GiB as Lambda functions, allowing customers to bring much larger code bases and sets of dependencies to Lambda. Supporting larger packages, while still meeting Lambda’s goals of rapid scale (adding up to 15,000 new containers per second for a single customer, and much more in aggregate), high request rate (millions of requests per second), high scale (millions of unique workloads), and low start-up times (as low as 50ms) presented a significant challenge. We describe the storage and caching system we built, optimized for delivering container images on-demand, and our experiences designing, building, and operating it at scale. We focus on challenges around security, efficiency, latency, and cost, and how we addressed these challenges in a system that combines caching, deduplication, convergent encryption, erasure coding, and block-level demand loading. Since building this system, it has reliably processed hundreds of trillions of Lambda invocations for over a million AWS customers, and has shown excellent resilience to load and infrastructure failures.
Research areas
Tags

Latest news

US, CA, Palo Alto
Global Optimization is a strategic initiative aimed at improving Amazon advertisers experience at global scale. We are looking for a passionate Applied Scientist to help pioneer the next generation of agentic AI applications for Amazon advertisers. In this role, you will design agentic architectures, develop tools and datasets, and contribute to building systems that can reason, plan, and act autonomously across complex advertiser workflows at global scale. You will work at the forefront of applied AI, developing methods for fine-tuning, reinforcement learning, and preference optimization, while helping create evaluation frameworks that ensure safety, reliability, and trust at scale. You will work backwards from the needs of advertisers—delivering customer-facing products that directly help them create, optimize, and grow their campaigns. Beyond building models, you will advance the agent ecosystem by experimenting with and applying core primitives such as tool orchestration, multi-step reasoning, and adaptive preference-driven behavior. This role requires working independently on ambiguous technical problems, collaborating closely with scientists, engineers, and product managers to bring innovative solutions into production. Key job responsibilities - Design and build agents that improve advertisers experiences globally - Design and implement advanced model and agent optimization techniques, including supervised fine-tuning, instruction tuning and preference optimization (e.g., DPO/IPO). - Design and implement optimization models that work at global scale taking into account nuances of multiple countries - Innovate new science models to help advertisers scale their campaigns globally - Curate datasets and tools for MCP. - Build evaluation pipelines for agent workflows, including automated benchmarks, multi-step reasoning tests, and safety guardrails. - Develop agentic architectures (e.g., CoT, ToT, ReAct) that integrate planning, tool use, and long-horizon reasoning. - Prototype and iterate on multi-agent orchestration frameworks and workflows. - Collaborate with peers across engineering and product to bring scientific innovations into production. - Stay current with the latest research in LLMs, RL, and agent-based AI, optimization and translate findings into practical applications. About the team The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through the latest generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. The Global Optimization team within Sponsored Products and Brands is focused on guiding and supporting 1.6MM advertisers to meet their advertising needs of creating and managing ad campaigns at global scale. At this scale, the complexity of diverse advertiser goals, campaign types, and market dynamics creates both a massive technical challenge and a transformative opportunity: even small improvements in guidance systems can have outsized impact on advertiser success and Amazon’s retail ecosystem. Our work is grounded in state-of-the-art agent architectures, tool integration, reasoning frameworks, and model customization approaches (including tuning, MCP, and preference optimization), ensuring our systems are both scalable and adaptive.
US, WA, Seattle
Amazon is seeking exceptional science talent to develop AI and machine learning systems that will enable the next generation of advanced manufacturing capabilities at unprecedented scale. We're building revolutionary software infrastructure that combines cutting-edge AI, large-scale optimization, and advanced manufacturing processes to create adaptive production control systems. As a Senior Research Scientist, you will develop and improve machine learning systems that enable real-time manufacturing flow decisions. You will leverage state-of-the-art optimization and ML techniques, evaluate them against representative manufacturing scenarios, and adapt them to meet the robustness, reliability, and performance needs of production environments. You will invent new algorithms where gaps exist. You'll collaborate closely with software engineering, manufacturing engineering, robotics simulation, and operations teams, and your outputs will directly power the systems that determine what to build next, where to allocate resources, and how to maximize throughput. The ideal candidate brings deep expertise in optimization and machine learning, with a proven track record of delivering scientifically complex solutions into production. You are hands-on, writing significant portions of critical-path scientific code while driving your team's scientific agenda. If you're passionate about inventing the intelligent manufacturing systems of tomorrow rather than optimizing those of today, this role offers the chance to make a lasting impact on the future of automation. Key job responsibilities - Identify and devise new scientific approaches for constraint identification, dispatch optimization, WIP release control, and predictive flow intelligence when the problem is ill-defined and new methodologies need to be invented - Lead the design, implementation, and successful delivery of scientifically complex solutions for real-time manufacturing flow optimization in production - Design and build ML models and optimization algorithms including constraint prediction, starvation risk forecasting, and dispatch optimization - Write a significant portion of critical-path scientific code with solutions that are inventive, maintainable, scalable, and extensible - Execute rapid, rigorous experimentation with reproducible results, closing the gap between simulation and real manufacturing environments - Build evaluation benchmarks that measure model performance against manufacturing outcomes including constraint utilization and throughput rather than traditional ML metrics alone - Influence your team's science and business strategy through insightful contributions to roadmaps, goals, and priorities - Partner with manufacturing engineering, robotics simulation, and applied intelligence teams to ensure scientific approaches are grounded in operational reality - Drive your team's scientific agenda and role model publishing of research results at peer-reviewed venues when appropriate and not precluded by business considerations - Actively participate in hiring and mentor other scientists, improving their skills and ability to deliver - Write clear narratives and documentation describing scientific solutions and design choices
US, WA, Seattle
At Amazon Selection and Catalog Systems (ASCS), our mission is to power the online buying experience for customers worldwide so they can find, discover, and buy any product they want. We innovate on behalf of our customers to ensure uniqueness and consistency of product identity and to infer relationships between products in Amazon Catalog to drive the selection gateway for the search and browse experiences on the website. We're solving a fundamental AI challenge: establishing product identity and relationships at unprecedented scale. Using Generative AI, Visual Language Models (VLMs), and multimodal reasoning, we determine what makes each product unique and how products relate to one another across Amazon's catalog. The scale is staggering: billions of products, petabytes of multimodal data, millions of sellers, dozens of languages, and infinite product diversity—from electronics to groceries to digital content. The research challenges are immense. GenAI and VLMs hold transformative promise for catalog understanding, but we operate where traditional methods fail: ambiguous problem spaces, incomplete and noisy data, inherent uncertainty, reasoning across both images and textual data, and explaining decisions at scale. Establishing product identities and groupings requires sophisticated models that reason across text, images, and structured data—while maintaining accuracy and trust for high-stakes business decisions affecting millions of customers daily. Amazon's Item and Relationship Platform group is looking for an innovative and customer-focused applied scientist to help us make the world's best product catalog even better. In this role, you will partner with technology and business leaders to build new state-of-the-art algorithms, models, and services to infer product-to-product relationships that matter to our customers. You will pioneer advanced GenAI solutions that power next-generation agentic shopping experiences, working in a collaborative environment where you can experiment with massive data from the world's largest product catalog, tackle problems at the frontier of AI research, rapidly implement and deploy your algorithmic ideas at scale, across millions of customers. Key job responsibilities Key job responsibilities include: * Formulate novel research problems at the intersection of GenAI, multimodal learning, and large-scale information retrieval—translating ambiguous business challenges into tractable scientific frameworks * Design and implement leading models leveraging VLMs, foundation models, and agentic architectures to solve product identity, relationship inference, and catalog understanding at billion-product scale * Pioneer explainable AI methodologies that balance model performance with scalability requirements for production systems impacting millions of daily customer decisions * Own end-to-end ML pipelines from research ideation to production deployment—processing petabytes of multimodal data with rigorous evaluation frameworks * Define research roadmaps aligned with business priorities, balancing foundational research with incremental product improvements * Mentor peer scientists and engineers on advanced ML techniques, experimental design, and scientific rigor—building organizational capability in GenAI and multimodal AI * Represent the team in the broader science community—publishing findings, delivering tech talks, and staying at the forefront of GenAI, VLM, and agentic system research
IN, KA, Bengaluru
Are you passionate about solving complex business problems at scale through Generative AI? Do you want to help build intelligent systems that reason, act, and learn from minimal supervision? If so, we have an exciting opportunity for you on Amazon's Trustworthy Shopping Experience (TSE) team. At TSE, our vision is to guarantee customers a worry-free shopping experience by earning their trust that the products they buy are safe, authentic, and compliant with regulations and policy. We do this in close partnership with our selling partners, empowering them with best-in-class tools and expertise to offer a high-quality, compliant selection that customers trust. As a Research Scientist I, you will bring subject matter expertise with fundamental improvements in at least one relevant discipline (e.g., NLP, computer vision, representation learning, agentic architecture) to contribute to next-generation agentic AI solutions that automate complex manual investigation processes at Amazon scale. You will invent, refine, and experiment with solutions spanning agentic reasoning, self-supervised representation learning, few-shot adaptation, multimodal understanding, and model compression. With guidance from senior scientists, you will stay current on research trends and benchmark your results against the state of the art. You will help design and execute experiments to identify optimal solutions, initiating the development and implementation of small components with team guidance. You will write secure, stable, testable, and well-documented production code at the level of an SDE I, rigorously evaluating models and quantifying performance. You will handle data in accordance with Amazon policies, troubleshoot issues to root cause, and ensure your work does not put the company at risk. Your scope of influence will typically be at the self-level, with the possibility of mentoring interns. You will participate in team design and prioritization discussions, learn the business context behind TSE's products, and escalate problems with proposed solutions. You will publish internal technical reports and may contribute to peer-reviewed publications and external review activities when aligned with business needs. This role offers a unique opportunity to contribute to end-to-end AI development—from research through production—with your contributions serving hundreds of millions of customers within months, not years. Key job responsibilities • Contribute to the design and development of agentic AI systems with multi-step reasoning, autonomous task execution, and multimodal intelligence, including feedback and memory mechanisms, leveraging reinforcement learning techniques for agent decision-making and policy optimization, with input and guidance from senior scientists • Develop novel models built on top of SFT (Supervised Fine-tuning) and RFT (Reinforced Fine-tuning) approaches, as well as few-shot approaches based on multimodal datasets spanning text, images, and structured data, applying mathematical optimization techniques to improve efficiency, resource allocation, and decision-making in complex workflows, working alongside senior scientists to identify optimal solutions • Contribute to building production-ready deep learning and conventional ML solutions, including multimodal fusion and cross-modal alignment techniques that seamlessly connect visual, textual, and relational understanding, to support automation requirements within your team's scope • Help identify customer and business problems; use reasonable assumptions, data, and customer requirements to solve well-defined scientific problems involving multimodal inputs such as unstructured text, documents, product images, and relational data, developing representations that capture complementary signals across modalities and mapping business goals to scientific metrics • May co-author research papers for peer-reviewed internal and/or external venues, including contributions in areas such as multimodal representation learning and vision-language modeling, and contribute to the wider scientific community by reviewing research submissions, when aligned with business needs • Prototype rapidly, iterate based on feedback, and deliver small components at SDE I level—including multimodal data pipelines and inference modules—that integrate into production-scale systems • Write secure, stable, testable, maintainable, and well-documented code, balancing model capability, deployment cost, and resource usage across multimodal architectures while understanding state-of-the-art data structures, algorithms, and performance tradeoffs • Rigorously test code and evaluate models across individual and combined modalities, quantifying their performance; troubleshoot issues, research root causes, and thoroughly resolve defects, leaving systems more maintainable • Participate in team design, scoping, and prioritization discussions through clear verbal and written communication; seek to learn the business context, science, and engineering behind your team's products, including how multimodal signals contribute to trust and safety decisions • Participate in engineering best practices with peer reviews; clearly document approaches and communicate design decisions; publish internal technical reports to institutionalize scientific learning • Help train and mentor scientist interns; identify and escalate problems with proposed solutions, taking ownership or ensuring clear hand-off to the right owner
US, NY, New York
We are seeking a scientist to further the development and application of analytics methods to examine the complex data flows of Amazon Ads and to translate deep-dives into actionable insights for our product teams. In this role you will develop new tools to analyze our advertising data to help improve the performance of our bidding algorithms, targeting and relevance systems, help advance our supply strategy, and evaluate the adoption and impact of feature releases. Key job responsibilities - Analyze data trends regarding supply, optimization, ad load, and advertising mix effects that affect advertiser performance and contribute to achieving advertiser goals - Present papers to senior leaders on issues like feature development impact on identity recognition rates, and changes of ad selection systems to improve fill rate highlighting insights that will inform our business development and engineering roadmaps - Formalize our analytics approach to Ads auctions by analyzing bid spreads, auction depth, and simulating impacts of potential auction structure changes - Identify, standardize, and operationalize KPIs to effectively measure the performance of all systems involved in ad serving, and use trend insights to inform business priorities - Partner with engineering teams to define data logging requirements and getting these prioritized in engineering roadmaps - Validate financial models through analysis - Develop and own ad revenue and supply intelligence analytics decks that provide ongoing deep-dives A day in the life The Ads Scientist will work closely with business leaders and engineers on developing common data architecture that will optimize our data logging at different grains, and will allow data interoperability from bid flow to optimization to campaign delivery. The scientist will then analyze the data and present papers and ongoing reports on actionable insights. About the team At Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team also puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work. Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future.
US, VA, Arlington
The People eXperience and Technology Central Science (PXTCS) team uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. The Benefits Science team is looking for an economist to transform complex business challenges into actionable scientific insights. In this role, you will partner directly with business leaders to design and evaluate pilots, build models using large-scale data, and scale successful prototypes into company-wide policies and programs. We're looking for someone who can combine rigorous scientific thinking with practical business acumen and is passionate about using economics to improve employee experiences at scale. The ideal candidate will thrive in interdisciplinary environments, working alongside engineers, data scientists, and business leaders from diverse backgrounds. Key job responsibilities - Design and conduct rigorous evaluations of benefits programs - Support the development and application of structural models - Develop experiments to evaluate the impact of benefits initiatives - Communicate complex findings to business stakeholders in clear, actionable terms - Work with engineering teams to develop scalable tools that automate and streamline evaluation processes A day in the life Work with teammates to apply economic methods to business problems. This might include identifying the appropriate research questions, writing code to implement a DID analysis or estimate a structural model, or writing and presenting a document with findings to business leaders. Our economists also collaborate with partner teams throughout the process, from understanding their challenges, to developing a research agenda that will address those challenges, to help them implement solutions.
US, WA, Seattle
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Our products are used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses a combination of econometrics, machine learning, and data science to build disruptive products for all our Advertising products. We also generate insights to guide Amazon Advertising strategy, providing direct support to senior leadership. We are looking for an experienced Economist who have a deep passion for building state-of-art causal models and ads measurement and optimization solutions, ability to communicate data insights and scientific vision, and execute strategic projects. As an Economist on this team, you will: - Lead the design and analysis of large-scale experiments to measure advertising effectiveness across Amazon's advertising products - Develop novel causal inference and econometric methodologies to solve attribution and incrementality measurement challenges at scale - Invent new optimization frameworks that translate measurement insights into actionable bidding, targeting, and budget allocation strategies for advertisers - Define the long-term science roadmap for ads measurement and optimization, identifying high-impact research directions and driving alignment across engineering, product, and science teams - Build and refine structural and reduced-form models that quantify the causal impact of advertising on consumer behavior, sales, and brand outcomes - Partner with engineering teams to operationalize econometric models into production systems serving millions of advertisers - Mentor and develop a team of economists and applied scientists, raising the bar on methodological rigor and scientific impact - Influence senior leadership through clear communication of complex economic concepts, shaping investment decisions and product strategy - Collaborate cross-functionally with product managers, engineers, and business leaders to translate business problems into well-defined economic questions with scalable solutions Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.
US, WA, Seattle
Interested in influencing what customers around the world see when they turn on Prime Video? The Prime Video Personalization and Discovery team matches customers with the right content at the right time, at all touch points throughout the content discovery journey. We are looking for a customer-focused, solutions-oriented Principal Data Scientist to develop next-gen measurement and experimentation systems within Prime Video Personalization and Discovery. You'll be part of an embedded science team driving projects across product and engineering teams that ultimately influence what millions of customers around the world see when the log into Prime Video. The ideal candidate brings experience building experiment-based measurement systems at scale, excellent stakeholder communication skills, and the ability to balance technical rigor with delivery speed and customer impact. You will build cross-functional support within Prime Video for high-quality, rigorous measurement, assess business problems, and support iterative scientific solutions that balance short-term delivery with long-term science roadmaps. Key job responsibilities - Define and drive the multi-year vision for experiment-based measurement systems within Prime Video - Partner with product stakeholders and science peers to identify strategic data-driven opportunities to improve the customer experience - Communicate findings, conclusions, and recommendations to technical and non-technical business leaders across Prime Video - Educate senior leaders about and advocate for high-quality measurement as an input to data-driven decisions - Mentor junior scientists and review technical artifacts to ensure quality - Stay up-to-date on the latest data science tools, techniques, and best practices and help evangelize them across the organization
US, WA, Seattle
Do you want to help shape the future of Amazon's physical retail presence? Worldwide Grocery Stores (WWGS), Location Strategy and Analytics team is looking for an Research Scientist to join us in developing advanced forecasting models, optimization models, and analytical tools to support critical real estate and store planning decisions for Amazon's Worldwide Grocery business, including Whole Foods Market. Our team is responsible for developing predictive models and tools to support Real Estate and Topology analysts in making important decisions regarding our stores—including new store openings, relocations, closures, remodels, design, new formats, and more. We leverage statistical modeling, machine learning, and GenAI to build solutions for store sales forecasting, sales transfer effects, macrospace optimization, store network optimization, store network diffusion planning, and causal effects. As a Research Scientist on our team, you will apply your technical and analytical skills to tackle complex business problems and develop innovative solutions to improve our forecasting and decision-making capabilities. You will collaborate with a diverse team of scientists, economists, and business partners to identify opportunities, develop hypotheses, build internal products, and translate analytical insights into actionable recommendations for Executive Leadership. Key job responsibilities - Design and implement forecasting models and machine learning solutions to predict store performance and optimize our retail network. - Analyze large datasets to uncover insights and patterns related to store performance, customer behavior, and market dynamics. - Develop end-to-end solutions, tools and frameworks to scale our ML model development and data analysis. - Leverage GenAI models to enhance user interaction with our solutions, improve overall user experience, and build new features. - Present research findings and recommendations to scientists, business leaders, and executives. - Collaborate with cross-functional teams to drive adoption of models and insights. - Stay current on latest developments in relevant fields and propose innovative approaches. About the team We are a team of scientists passionate about leveraging data and advanced analytics to drive strategic decisions for Amazon's grocery business. Our work directly impacts Amazon's worldwide grocery store growth and development strategy. We foster a collaborative environment where team members are encouraged to think creatively, challenge assumptions, and pursue novel approaches to solving complex problems. Our team is at the forefront of applying a multitude of techniques - including GenAI - to improve our scientific solutions and products.
US, WA, Bellevue
Have you ever ordered a product on Amazon and when that box with the smile arrived, wondered how it got to you so fast? Wondered where it came from and how much it cost Amazon? If so, the Amazon Global Supply Chain Optimization Technology (SCOT) organization is for you. Watch this video to learn more about our organization, SCOT: http://bit.ly/amazon-scot We are the Optimal Sourcing Systems team (OSS) within SCOT and are looking for a Data Scientist II to join us! OSS designs and builds systems that measure and manage Amazon’s supplier capabilities, identify and react to supply disruptions, and prioritizes inbound freight for our global network. OSS software is used by every country Amazon services, and is a critical link to ensuring Amazon offers the products our customers want, at the lowest possible cost. This team under OSS orchestrates and tracks inventory movement into Amazon's network, maintains performance feedback loops, and ensures vendor compliance. The Data Scientist II, in partnership with the Product Management, Operations, and Tech teams, will lead efforts in four areas: 1) Building models to set optimal parameters such as lead times to ensure the accuracy of our Inbound network 2) Building analytical frameworks to identify and drive improvements in purchase order lifecycle management and defect coaching/chargebacks 3) Developing Gen AI solutions related to dispute evaluation and vendor coaching 4) Building models and solutions to enable collaborative inventory planning with vendors The ideal candidate thrives in ambiguous problem spaces, relishes working with large volumes of data, and enjoys the challenge of highly complex supply chain contexts. They can translate complex business logic into scalable models and communicate insights effectively to both technical and non-technical stakeholders. Keys to success in this role include exceptional analytics, statistics, judgment, and communication skills. Experience with supply chain optimization, operations research, or vendor management systems is a plus. Key job responsibilities - Collaborate with product managers, science, and engineering teams to design and implement model solutions for Sourcing Execution & Performance systems - Use large datasets or experiments to make causal inferences or predictions - Work with engineers to automate science analysis processes and build scalable measurement solutions - Interpret data, write reports, and make actionable recommendations - Drive technical standards and best practices for the team's Science solutions - Mentor and provide technical guidance to other team members on complex projects A day in the life Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: - Medical, Dental, and Vision Coverage - Maternity and Parental Leave Options - Paid Time Off (PTO) - 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply!