Social
Publication

Rethinking zero-shot video classification: end-to-end training for realistic applications

By Biagio Brattoli, Joe Tighe, Fedor Zhdanov, Pietro Perona, Krzysztof Chalupka
2020
Download Copy BibTeX GitHub
Share
Download
Copy BibTeX
GitHub
Share
Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes.However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin.
Research areas
Tags

Latest news

Work with us

See more jobs See more jobs
Applied Scientist - LLM, Alexa
GB, Cambridge
Alexa is looking for an Applied Scientist with a strong background in Natural Language Processing (NLP) and Large Language Models (LLMs) to help build state-of-the-art conversational systems. In this role, you will collaborate with a large team of scientists training the Large Language Models that power the Alexa stack, as well as software engineers serving them in production systems. You will own solutions end-to-end: from ideation and research through to production deployment, enabling conversational assistants to support external tools, leverage diverse sources of information, and deliver novel reasoning capabilities to millions of Alexa customers. Key job responsibilities As an Applied Scientist, you will develop innovative solutions to complex problems to extend the functionalities of conversational assistants. You will use your technical expertise to research and implement novel algorithms and modelling solutions in collaboration with other scientists and engineers. You will analyze customer behaviors and define metrics to enable the identification of actionable insights and measure improvements in customer experience. You will communicate results and insights to both technical and non-technical audiences through written reports, presentations and external publications. You would be able to bi-modal on science and engineering: someone who combines strong scientific foundations with the execution skills to ship high-quality solutions. A day in the life As an Applied Scientist on the Alexa Science team, you'll drive innovation in evaluating new product experiences while discovering novel approaches to enhance model capabilities and enrich customer interactions. You'll collaborate with cross-functional teams of engineers and scientists to identify root causes of model and system integration issues, continuously improving the end-to-end customer experience. You'll partner closely with scientists developing and fine-tuning large language models, engineers building low-latency inference infrastructure, and product teams defining customer experience metrics. About the team We are a team of applied scientists and engineers building the intelligence layer that powers Alexa+. Our work sits at the intersection of large language models, decision-making under uncertainty, and production ML systems. What we build directly shapes the customer experience: determining which models serve their requests, optimizing response latency, and creating natural, seamless interactions. We're a collaborative team that values rigorous experimentation, clear communication, and delivering solutions that perform at scale in real-world environments.
Applied Scientist II, Perception
US, CA, San Francisco
Amazon is on a mission to redefine the future of automation — and we're looking for exceptional talent to help lead the way. We are building the next generation of advanced robotic systems that seamlessly blend cutting-edge AI, sophisticated control systems, and novel mechanical design to create adaptable, intelligent automation solutions capable of operating safely alongside humans in dynamic, real-world environments. At Amazon, we leverage the power of machine learning, artificial intelligence, and advanced robotics to solve some of the most complex operational challenges at a scale unlike anywhere else in the world. Our fleet of robots spans hundreds of facilities globally, working in sophisticated coordination to deliver on our promise of customer excellence — and we're just getting started. As a Applied Scientist in Robot Perception, you will be at the forefront of this transformation. You will develop and deploy state-of-the-art perception algorithms that enable robots to truly understand and interact with the physical world — bridging the gap between theoretical research and real-world impact. Bringing deep expertise in Computer Vision and a nuanced understanding of the capabilities and limitations of modern Vision-Language Models (VLMs), you will innovate boldly and push the boundaries of what's possible. Our vision for the Perception layer is ambitious: to enable seamless, intelligent interaction between the user, the robot, and its environment. This is a rare opportunity to work at the intersection of deep learning, large language models, and robotics — contributing to research that doesn't just advance the field, but reshapes it. You will collaborate with world-class teams pioneering breakthroughs in dexterous manipulation, locomotion, and human-robot interaction, all at an unprecedented scale. Join us in building intelligent robotic systems that will define the future of automation and human-robot collaboration. Key job responsibilities - Design, develop, and deploy perception algorithms for robotics systems, including object detection, segmentation, tracking, depth estimation, and scene understanding - Lead research initiatives in computer vision, sensor fusion and 3D perception - Collaborate with cross-functional teams including robotics engineers, software engineers, and product managers to define and deliver perception capabilities - Drive end-to-end ownership of ML models — from data collection and labeling strategy to training, evaluation, and deployment - Mentor junior scientists and engineers; contribute to a culture of technical excellence - Define and track key metrics to measure perception system performance in real-world environments - Publish research findings in top-tier venues (CVPR, ICCV, ECCV, ICRA, NeurIPS, etc.) and contribute to patents A day in the life - Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment - Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations - Actively contribute to brainstorming sessions on adjacent topics, bringing fresh perspectives that help peers grow and succeed — and in doing so, build lasting trust across the team - Mentor team members while maintaining significant hands-on contribution to technical solutions
Applied Scientist, Machine Learning Accelerator - Annapurna Labs
US, TX, Austin
Our team is involved with pre-silicon design verification for custom IP. A critical requirement of the verification flow is the requirement of legal and realistic stimulus of a custom Machine Learning Accelerator Chip. Content creation is built using formal methods that model legal behavior of the design and then solving the problem to create the specific assembly tests. The entire frame work for creating these custom tests is developed using a SMT solver and custom software code to guide the solution space into templated scenarios. This highly visible and innovative role requires the design of this solving framework and collaborating with design verification engineers, hardware architects and designers to ensure that interesting content can be created for the projects needs. Key job responsibilities Develop an understanding for a custom machine learning instruction set architecture. Model correctness of instruction streams using first order logic. Create custom API's to allow control over scheduling and randomness. Deploy algorithms to ensure concurrent code is safely constructed. Create coverage metrics to ensure solution space coverage. Use novel methods like machine learning to automate content creation. About the team Utility Computing (UC) AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps services in AWS, including support for customers who require specialized security solutions for customers who require specialized security solutions for their cloud services. Annapurna Labs (our organization within AWS UC) designs silicon and software that accelerates innovation. Customers choose us to create cloud solutions that solve challenges that were unimaginable a short time ago—even yesterday. Our custom chips, accelerators, and software stacks enable us to take on technical challenges that have never been seen before, and deliver results that help our customers change the world.
Member of Technical Staff, Multimodal Agents, AGI Autonomy
US, CA, San Francisco
Amazon AGI Lab is a frontier research and product team combining the speed of a startup with Amazon’s scale and resources. We build agents that can perceive, reason, and take action to complete real-world tasks. The lab is designed to empower AI researchers and engineers to make major breakthroughs with speed and focus toward this goal. Each team in the lab has the autonomy to move fast and the long-term commitment to pursue high-risk, high-payoff research. We're hiring a principal engineer who can take models from prototype to production and build the systems that make them run reliably at scale. The bar is end-to-end ownership: your work can range from working alongside researchers to build novel architectures, to being the person who decides what the agent runtime looks like, where the data lives, and how we know it's delivering value. Key job responsibilities - Set the technical direction for the team - Partner closely with researchers to take emerging VLM and agent ideas from prototype to robust, instrumented systems that can be evaluated, improved, and scaled - Create tooling that accelerates research and engineering velocity - Raise the engineering bar for the team through technical design reviews, mentoring, principled architecture, high-quality code, observability, and operational excellence - Influence the broader AGI organization by identifying reusable primitives, writing clear technical strategy, and creating systems that other teams can build on - Be a thought leader & represent the lab externally by sharing ideas through thoughtful writing, conference talks, research publications, and open-source contributions, helping advance the field while raising the visibility and impact of the team’s work
Applied Scientist, Amazon Precision Match
US, WA, Seattle
We are seeking an Applied Scientist to join the Amazon Precision Match (APM) team within Customer Journey, Network Solutions. APM is a transformative initiative replacing Amazon's legacy queue-based customer service routing with intelligent algorithmic matching — connecting customers with the best available service option based on their needs and Customer Service Associates (CSA) capabilities. This role will drive the science behind a high-scale system with significant projected impact on operational efficiency and customer experience. You will work at the intersection of recommendation systems, real-time ML inference, and large-scale experimentation to redefine how Amazon serves its customers. Key job responsibilities - Design, develop, and optimize ML-based matching algorithms that pair customers with optimal CSAs based on contact complexity, intent, and CSA skill profiles. - Build and iterate on feature engineering pipelines across CSA-level (skills, tenure, sentiment handling), contact-level (intent, complexity, urgency), and customer-level (language, communication style) attributes. - Run offline simulations on large-scale historical contact data and design statistically rigorous A/B experiments to validate matching improvements. - Develop real-time low-latency scoring and inference systems for production contact routing. - Address the cold start problem for new CSAs and build continuous model retraining infrastructure using production feedback. - Partner with CS Economics, Capacity Planning, and Quality teams on experiment design and results interpretation. - Evolve the matching framework from individual CSA ranking to set-based optimization balancing performance and operational sustainability. A day in the life You will spend your days iterating on matching models, analyzing experiment results from live production traffic, and collaborating with engineers and product managers to translate science insights into system improvements. You'll partner with the Customer Service Economics team to design experiments, review simulation outputs, and present findings to senior leadership. You'll also deep-dive into CSA behavioral patterns, contact transcripts, and performance data to identify new matching signals and continuously improve the algorithm. About the team The Amazon Precision Match team is a high-impact, fast-moving science and engineering team within Customer Journey, Network Solutions. Our mission is to ensure every Amazon customer is connected with the right service option at the right time — improving customer experience while driving operational efficiency at scale. We value intellectual curiosity, rigorous experimentation, and a bias for action. We operate with a continuous improvement flywheel: offline simulation, A/B testing, and production rollout. We collaborate closely with Customer Service Operations, Capacity Planning, Quality, and partner science teams across Amazon.
Applied Scientist, Pricing Science
US, WA, Seattle
Amazon's Pricing Science is seeking a driven Applied Scientist to harness planet scale multi-modal datasets, and navigate a continuously evolving competitor landscape, in order to regularly generate fresh customer-relevant prices on billions of Amazon products worldwide. We are looking for a talented, organized, and customer-focused applied researchers to join our Pricing Optimization science group, with a charter to measure, refine, and launch customer-obsessed improvements to our pricing algorithms across all products listed on Amazon. This role requires an individual with exceptional machine learning and predictive modeling skills, causal and experimental evaluation experience, excellent cross-functional collaboration skills and business acumen, and an entrepreneurial spirit. We are looking for an experienced innovator, who is a self-starter, comfortable with ambiguity, demonstrates strong attention to detail, and has the ability to work independently to deliver business impact. Key job responsibilities - See the big picture. Understand and develop science to influence the long term vision for Amazon's science-based competitive, perception-preserving pricing techniques - Build strong collaborations. Partner with product, engineering, and data teams within Pricing & Promotions to deploy models at Amazon scale - Stay informed. Establish mechanisms to stay up to date on latest scientific advancements in machine learning, reinforcement learning, causal ML, and multi-objective optimization techniques. Identify opportunities to apply them to relevant Pricing & Promotions business problems - Keep innovating for our customers. Foster an environment that promotes rapid experimentation, continuous learning, and incremental value delivery. - Successfully execute & deliver. Apply your exceptional technical machine learning expertise to incrementally move the needle on some of our hardest pricing problems. A day in the life We are hiring an applied scientist to drive our pricing optimization initiatives. The Price Optimization science team drives cross-domain and cross-system improvements through: - Invent and deliver price optimization, simulation, and competitiveness tools for Sellers. - Promotion optimization initiatives exploring CX, discount amount, and cross-product optimization opportunities. - Identifying opportunities to optimally price across systems and contexts (marketplaces, request types, event periods) Price is a highly relevant input into many partner-team architectures, and is highly relevant to the customer, therefore this role creates the opportunity to drive extremely large impact (measured in Bs not Ms), but demands careful thought and clear communication. About the team About the team: the Pricing Optimization team within P2 Science owns price quality, discovery and discount optimization initiatives, including criteria for internal price matching, price discovery into search, p13N and SP, pricing bandits, and Promotion type optimization. We leverage planet scale data on billions of Amazon and external competitor products to build advanced optimization models for pricing, elasticity estimation, product substitutability, and optimization. We preserve long term customer trust by ensuring Amazon's prices are always competitive and error free.
Senior Applied Scientist , Sponsored Products
US, NY, New York
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. We are seeking a technical leader for our Supply Science team. This team is within the Sponsored Product team, and works on complex engineering, optimization, econometric, and user-experience problems in order to deliver relevant product ads on Amazon search and detail pages world-wide. The team operates with the dual objective of enhancing the experience of Amazon shoppers and enabling the monetization of our online and mobile page properties. Our work spans ML and Data science across predictive modeling, reinforcement learning (Bandits), adaptive experimentation, causal inference, data engineering. Key job responsibilities Search Supply and Experiences, within Sponsored Products, is seeking a Senior Applied Scientist to join a fast growing team with the mandate of creating new ads experience that elevates the shopping experience for our hundreds of millions customers worldwide. We are looking for a top analytical mind capable of understanding our complex ecosystem of advertisers participating in a pay-per-click model– and leveraging this knowledge to help turn the flywheel of the business. As a Senior Applied Scientist on this team you will: --Act as the technical leader in Machine Learning and drive full life-cycle Machine Learning projects. --Lead technical efforts within this team and across other teams. --Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production. --Run A/B experiments, gather data, and perform statistical analysis. --Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. --Work closely with software engineers to assist in productionizing your ML models. --Research new machine learning approaches. --Recruit Applied Scientists to the team and act as a mentor to other scientists on the team. A day in the life The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail, and with an ability to work in a fast-paced, high-energy and ever-changing environment. The drive and capability to shape the direction is a must. About the team We are a customer-obsessed team of engineers, technologists, product leaders, and scientists. We are focused on continuous exploration of contexts and creatives where advertising delivers value to customers and advertisers. We specifically work on new ads experiences globally with the goal of helping shoppers make the most informed purchase decision. We obsess about our customers and we are continuously innovating on their behalf to enrich their shopping experience on Amazon
Data Scientist, Seller Fee Science
IN, KA, Bengaluru
The Seller Fee Science Team integrates economic modeling, machine learning, and artificial intelligence to guide fee strategy, quantify its impact, and ensure fees are accurately computed and explained for billions of transactions between Amazon selling partners and customers. We help build the foundations for growing selling partner businesses, bringing the best selection and prices to Amazon customers, and helping Amazon leaders make and implement high impact decisions that optimally balance profitability and growth. Our team brings together world-class economists, physicists, mathematicians, and computer scientists to tackle diverse challenging problems that require theoretical rigor and deliver real-world impact. As an data scientist on our team, this role will focus on the application of data analysis, econometrics, machine learning, and artificial intelligence to measure and predict Amazon's P&L, with emphasis on fee revenue. This blends the tools of data science, statistics, and ML/AI. Your work will shape not only how fees are decided, but how they are interpreted and planned. We are seeking scientists who are motivated by first principles, disciplined experimentation, and the technical challenge of deploying ideas at global scale. This is an opportunity to work on consequential problems where analytic rigor meets real-world complexity, and where your analysis, models, algorithms, and systems will directly influence the experience of millions of sellers. If you are driven to build elegant solutions to hard problems—and to see them operate in production at meaningful scale—we would welcome the opportunity to build with you. Key job responsibilities ** Translate ambiguous business challenges into well-defined scientific problems with measurable impact. ** Identify opportunities to improve fee revenue measurement, prediction, planning, structure, and level. ** Identify opportunities to improve measurement, and prediction of other items of the P&L, at appropriate levels of granularity. ** Design, develop, and deploy econometric or AI/ML models that improve our understanding of the relationship between fees and costs, or predict fee revenue, and other elements of the P&L. ** Partner closely with finance and fee strategy teams to formulate scientific questions, communicate results, and productionalize solutions. **Apply rigorous simulation methods to validate models and quantify business impact at scale. **Communicate scientific innovations and results clearly to cross-functional stakeholders and contribute to the broader internal and external scientific community through publications, talks, and technical artifacts. About the team Amazon’s third-party marketplace is a multibillion-dollar global service, connecting customers and sellers across through billions of transactions annually. The Seller Fee Science Team integrates economic modeling, machine learning, and artificial intelligence to guide business fee strategy, ensure fees are accurately computed for millions of products, and improve the seller experience with AI tools that support any fee related contact (understanding, audit, and dispute). We build the scientific foundation that empowers sellers to grow their businesses with clarity and confidence. Our team brings together world-class economists, physicists, mathematicians, and computer scientists to tackle diverse challenging problems that require theoretical rigor and deliver real-world impact.
Director, Applied Science, Full Funnel Advertising (FAIM)
US, CA, Palo Alto
We're seeking an Applied Science leader to build AI/ML-powered agentic systems that operate across the full advertising funnel, from awareness through conversion, autonomously optimizing advertiser outcomes at scale. You'll lead a world-class science and engineering team that ships production systems leveraging models and multi-agent architectures, transforming how millions of customers discover products and how advertisers engage with Amazon Ads powered by AI. You'll set the bar for technical excellence and high-velocity innovation: attract and retain top talent, maintain operational excellence, and ensure research translates into measurable, customer-centric impact. Key job responsibilities * Lead the development and implementation of generative AI strategies for Full funnel campaigns and New product campaigns * Drive technical strategy and roadmap decisions that balance innovation, scalability, and customer impact * Drive the architecture and delivery of production-grade multi-agent systems, including planning agents, bidding agents, creative agents, and measurement agents * Collaborate with cross-functional teams to integrate advanced AI technologies into existing advertising platforms * Spearhead research and innovation in AI-powered advertising solutions * Build and develop cross-functional teams of applied scientists and engineers * Make critical build-vs-buy and architectural tradeoff decisions across the agentic stack A day in the life Your day will be a dynamic blend of strategic leadership, technical innovation, and collaborative problem-solving. You'll work closely with cross-functional teams to design and implement advanced AI technologies that enhance advertising experiences, driving meaningful connections between brands and customers. About the team We are a passionate group of innovators dedicated to developing AI powered advertiser products that balance the needs of advertisers and enhance the user experience. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising.
Quantum Applied Scientist, Processor Test & Measurement, Amazon Center for Quantum Computing
US, CA, Pasadena
The Amazon Center for Quantum Computing in Pasadena, CA, is looking to hire an Applied Scientist in the Processor Test and Measurement group. You will join a multi-disciplinary team of theoretical and experimental physicists, materials scientists, and hardware and software engineers working at the forefront of quantum computing. This role focuses on the verification and validation of the circuit components that make up a quantum error correction (QEC) code — such as gates, reset, and readout — and on understanding how the performance of those components contributes to overall QEC performance. We are looking for someone who enjoys connecting component-level measurements to integrated system behavior, and who is motivated by working across teams to understand it. Much of the work involves partnering with processor design, theory, and QEC colleagues to validate that new devices behave as their Hamiltonians predict, and to explore the gaps when they don't. A comfort with error budgeting — reasoning about where component performance comes from and what limits it — is central to the role. Candidates with a track record of original scientific contributions will be preferred. We value strong engineering principles, resourcefulness, problem solving, and clear communication, along with the ability to work effectively within a team. As an Applied Scientist you will have the opportunity to pursue new ideas and stay abreast of the field of experimental quantum computation. Key job responsibilities We are looking to hire an Applied Scientist to help verify and validate the circuit components of our error-corrected quantum processors and to understand how their performance maps to QEC requirements. Depending on background and interest, the work may include: - Collaborating with theory and processor design teams to develop experimental test plans that validate new processor designs and check that fabricated devices meet their intent. - Characterizing the building blocks of a QEC code and building error budgets that explain and bound their performance. - Designing experiments that help separate effects such as crosstalk and spectator interactions from intrinsic component performance. - Prototyping calibration and measurement approaches that can later be matured for automated, large-scale processor bring-up and QEC demonstrations. - Investigating discrepancies between measured and expected behavior, and feeding what you learn back into design and theory. You will have the opportunity to take part in high-impact research projects that intersect with our engineering roadmap, working closely with processor, theory, and QEC stakeholders so that component-level decisions are informed by overall system performance. A day in the life About the team Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps services in AWS. Within AWS UC, Amazon Dedicated Cloud (ADC) roles engage with AWS customers who require specialized security solutions for their cloud services. Inclusive Team Culture AWS values curiosity and connection. Our employee-led and company-sponsored affinity groups promote inclusion and empower our people to take pride in what makes us unique. Our inclusion events foster stronger, more collaborative teams. Our continual innovation is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do. Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Export Control Requirement: Due to applicable export control laws and regulations, candidates must be either a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum, or be able to obtain a US export license. If you are unsure if you meet these requirements, please apply and Amazon will review your application for eligibility.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social