Social
Computer vision

A quick guide to Amazon’s papers at CVPR 2024

As in other areas of AI, generative models and foundation models — such as vision-language models — are a hot topic.

By Staff writer
June 13, 2024
4 min read
Share

In the past few years, foundation models and generative-AI models — and particularly, large language models (LLMs) — have become a major topic of AI research. That’s true even in field of computer vision, with its increased focus on vision-language models that yoke LLMs and image encoders.

This shift can be seen in the topics of the Amazon papers accepted to this year’s Computer Vision and Pattern Recognition Conference (CVPR 2024). A plurality of the papers deal with vision-language models, while a number of others concern related topics such as visual question answering, hallucination mitigation, and retrieval-aided generation. At the same time, however, classical computer vision topics such as 3-D reconstruction, object tracking, and pose estimation remain well represented.

3-D reconstruction

No more ambiguity in 360◦ room layout via bi-layout estimation
Yu-Ju Tsai, Jin-Cheng Jhang, Jingjing Zheng, Wei Wang, Albert Chen, Min Sun, Cheng-Hao Kuo, Ming-Hsuan Yang

ViewFusion: Towards multi-view consistency via interpolated denoising
Xianghui Yang, Yan Zuo, Sameera Ramasinghe, Loris Bazzani, Gil Avraham, Anton van den Hengel

Multiview consistency.png
The object views produced by standard diffusion models are often realistic, but adjacent views may lack alignment (left). ViewFusion incorporates an autoregressive process that fosters consistency across views (right). From "ViewFusion: Towards multi-view consistency via interpolated denoising".

Algorithmic information theory

Interpretable measures of conceptual similarity by complexity-constrained descriptive auto-encoding
Alessandro Achille, Greg Ver Steeg, Tian Yu Liu, Matthew Trager, Carson Klingenberg, Stefano Soatto

Geospatial analysis

Bridging remote sensors with multisensor geospatial foundation models
Boran Han, Shuai Zhang, Xingjian Shi, Markus Reichstein

Hallucination mitigation

Multi-modal hallucination control by visual information grounding
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, Stefano Soatto

THRONE: An object-based hallucination benchmark for the free-form generations of large vision-language models
Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, C. J. Taylor, Stefano Soatto

Metric learning

Learning for transductive threshold calibration in open-world recognition
Qin Zhang, Dongsheng An, Tianjun Xiao, Tong He, Qingming Tang, Ying Nian Wu, Joe Tighe, Yifan Xing, Stefano Soatto

Model robustness

GDA: Generalized diffusion for robust test-time adaptation
Yun Yun Tsai, Fu-Chen Chen, Albert Chen, Junfeng Yang, Che-Chun Su, Min Sun, Cheng-Hao Kuo

Object-centric learning

Adaptive slot attention: Object discovery with dynamic slot number
Ke Fan, Zechen Bai, Tianjun Xiao, Tong He, Max Horn, Yanwei Fu, Francesco Locatello, Zheng Zhang

Object tracking

Self-supervised multi-object tracking with path consistency
Zijia Lu, Bing Shuai, Yanbei Chen, Zhenlin Xu, Davide Modolo

Pose estimation

MRC-Net: 6-DoF pose estimation with multiscale residual correlation
Yuelong Li, Yafei Mao, Raja Bala, Sunil Hadap

Pose estimation.png
Image pairs in which the left image is a camera image and the right image superimposes colorized 3-D models of objects, with estimated six-degree-of-freedom poses, on the original image.

Responsible AI

FairRAG: Fair human generation via fair retrieval augmentation
Robik Shrestha, Yang Zou, James Chen, Zhiheng Li, Yusheng Xie, Tiffany Deng

Retrieval-augmented generation

CPR: Retrieval augmented generation for copyright protection
Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, Stefano Soatto

Security

Sharpness-aware optimization for real-world adversarial attacks for diverse compute platforms with enhanced transferability
Muchao Ye, Xiang Xu, Qin Zhang, Jon Wu

Video-language models

VidLA: Video-language alignment at scale
Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan, Son Tran, Benjamin Yao, Belinda Zeng, Mubarak Shah, Trishul Chilimbi

Vision-language models

Accept the modality gap: An exploration in the hyperbolic space
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, Ajanthan Thalaiyasingam

Modality gap.png
"Accept the modality gap: An exploration in the hyperbolic space" propose a new angle-based contrastive loss that permits the placement of images anywhere along the axis emanating from a text embedding, enabling a hierarchy among images.

Enhancing vision-language pre-training with rich supervisions
Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Vijay Mahadevan, Zhuowen Tu, Stefano Soatto

GROUNDHOG: Grounding large language models to holistic segmentation
Yichi Zhang, Martin Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi (QZ) Gao, Joyce Chai

Hyperbolic learning with synthetic captions for open-world detection
Fanjie Kong, Yanbei Chen, Jiarui Cai, Davide Modolo

Non-autoregressive sequence-to-sequence vision-language models
Kunyu Shi, Qi Dong, Luis Goncalves, Zhuowen Tu, Stefano Soatto

On the scalability of diffusion-based text-to-image generation
Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R. Manmatha, Ashwin Swaminathan, Zhuowen Tu, Stefano Ermon, Stefano Soatto

UNet scaling.png
The effect of UNet scaling on text-image alignment. In "On the scalability of diffusion-based text-to-image generation", Amazon researchers vary a UNet along two dimensions: channel number (left) and transformer depth (right). The prompts are (1) "square blue apples on a tree with circular yellow leaves"; (2) "five frosted glass bottles"; (3) "a yellow box to the right of a blue sphere"; (4) "the International Space Station flying in front of the moon".

Visual question answering

GRAM: Global reasoning for multi-page VQA
Tsachi Blau, Sharon Fogel, Roi Ronen, Alona Golts, Roy Ganz, Elad Ben Avraham, Aviad Aberdam, Shahar Tsiper, Ron Litman

Question aware vision transformer for multimodal reasoning
Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, Ron Litman

Synthesize step-by-step: Tools, templates and LLMs as data generators for reasoning-based chart VQA
Zhuowan Li, Bhavan Jasani, Peng Tang, Shabnam Ghadar

Research areas
Tags
About the Author
Staff writer

Related content

Work with us

See more jobs See more jobs
Applied Scientist - Perception (SLAM/VIO), Fauna
US, NY, New York
We are seeking an Applied Scientist to develop and optimize Visual Inertial Odometry (VIO) and sensor fusion systems for our intelligent robots. In this role, you will design, implement, and deploy state estimation and tracking algorithms that enable robots to understand their position and motion in real time, even in challenging and dynamic environments. You will own the full pipeline from algorithm development through embedded deployment, ensuring that perception systems run efficiently on resource-constrained robotic hardware. You will also leverage modern machine learning approaches to push the boundaries of classical perception methods, combining learned representations with geometric techniques to achieve robust, real-time performance. This is a deeply hands-on role. You will work directly with sensors, hardware, and real-world data, while prototyping, testing, and iterating in physical environments. The ideal candidate has strong foundations in VIO and sensor fusion, practical experience optimizing algorithms for embedded platforms, and familiarity with how modern deep learning is transforming perception. Key job responsibilities - Design and implement Visual Inertial Odometry algorithms for robust real-time state estimation on robotic platforms like Sprout - Develop multi-sensor fusion pipelines integrating cameras, IMUs, and other sensing modalities for accurate pose tracking - Optimize perception and tracking algorithms for deployment on embedded hardware (e.g., ARM, GPU-accelerated edge devices) under strict latency and power constraints - Apply modern ML-based perception techniques (learned features, depth estimation, neural odometry) to complement and improve classical geometric approaches - Build and maintain calibration, evaluation, and benchmarking infrastructure for perception systems - Collaborate with hardware, controls, and navigation teams to integrate perception outputs into the robot’s autonomy stack - Lead technical projects from research prototyping through production deployment
Sr. Applied Scientist, Special Projects
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the limits. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. As an Applied Scientist on our team, you will focus on building state-of-the-art ML models for biology. Our team rewards curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the forefront of both academic and applied research in this product area, you have the opportunity to work together with a diverse and talented team of scientists, engineers, and product managers and collaborate with other teams. Key job responsibilities - Build, adapt and evaluate ML models for life sciences applications - Collaborate with a cross-functional team of ML scientists, biologists, software engineers and product managers
Economist III - AMZ9898444
US, MA, Boston
MULTIPLE POSITIONS AVAILABLE Employer: AMAZON.COM SERVICES LLC Offered Position: Economist III Job Location: Boston, Massachusetts Job Number: AMZ9898444 Position Responsibilities: Mentor and guide the applied scientists and economists in our organization and hold us to a high standard of technical rigor and excellence in science. Design and lead roadmaps for complex science projects to help SP have a delightful selling experience while creating long term value for our shoppers. Work with our engineering partners and draw upon your experience to meet latency and other system constraints. Identify untapped, high-risk technical and scientific directions, and simulate new research directions that you will drive to completion and deliver. Be responsible for communicating our science innovations to the broader internal & external scientific community. Position Requirements: Ph.D. or foreign equivalent degree in Economics or a related field and two years of research or work experience in the job offered or a related occupation. Must have two years of research or work experience in the following skill(s): 1) experience in econometrics including experience with program evaluation, forecasting, time series, panel data, or high dimensional problems; 2) experience with economic theory and quantitative methods; and 3) coding in a scripting language such as R, Python, or similar. Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation. 40 hours / week, 8:00am-5:00pm, Salary Range $159,200/year to $215,300/year. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits.#0000
Principal Applied Scientist, Automated Reasoning Group
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues.
Sr. Research Scientist, Pricing Science
US, WA, Seattle
Amazon's Worldwide Pricing & Promotions organization is seeking a talented, hands-on Research Scientist to join the Pricing and Promotion Optimization Science (P2OS) team — the optimization "application layer" within Amazon's Pricing Sciences organization. Amazon adjusts prices on hundreds of millions of products daily across a global marketplace; P2OS is the team that makes those prices optimal. P2OS is a small, specialized unit with an outsized charter: develop and maintain the models that determine optimal prices and promotions across Amazon's catalog and merchant programs. We own the full optimization stack — from price prediction to promotion targeting to competitiveness guardrails — and we measure success in terms of accretive Gross Contribution and Customer Pricing Perception (GCCP). Our work spans Retail Core, Amazon Business, Fresh, Grocery, and international marketplaces, and we are continually investing in more extensible, generalizable science foundations to keep pace with a growing and evolving business. We are looking for an innovative, organized, and customer-focused scientist with exceptional machine learning and predictive modeling skills, causal and experimental evaluation experience, and the entrepreneurial spirit to apply state-of-the-art methods to some of the most impactful pricing problems in e-commerce. You should be comfortable with ambiguity, motivated by measurable business impact, and excited by the opportunity to work at Amazon-scale. Key job responsibilities * Innovate and build. Design, develop, and deploy machine learning models that set optimal prices and promotions across Amazon's global catalog. Own models end-to-end — from problem formulation and data analysis through offline evaluation, A/B testing, and production launch. * Build a generalizable science foundation. Develop models and evaluation frameworks designed to scale across merchant programs, product categories, and marketplaces — enabling cross-learning and reducing the time and cost of applying science to new business contexts. * Build and evolve optimization systems. Design and improve optimization systems — including reinforcement learning and multi-objective optimization approaches — that automate price and promotion decisions at scale across millions of products. * Apply generative AI and foundation models. Identify and pursue opportunities to leverage large language models, embeddings, and generative AI techniques in pricing science — from enriching product representations and extracting competitive signals from unstructured data, to building more capable and explainable pricing systems. * Experiment rigorously. Design and execute A/B tests and causal inference studies to measure the business and customer impact of pricing model changes. Translate findings into production-ready science improvements. * Stay at the frontier. Establish mechanisms to track the latest advances in reinforcement learning, causal ML, multi-objective optimization, generative AI, and demand modeling — and identify opportunities to apply them to Pricing & Promotions business problems. * See the big picture. Contribute to the long-term scientific vision for how Amazon sets competitive, perception-preserving prices — balancing profitability, customer trust, and marketplace health.
Applied Scientist, Navigation
US, CA, San Francisco
Amazon is on a mission to redefine the future of automation — and we're looking for exceptional talent to help lead the way. We are building the next generation of advanced robotic systems that seamlessly blend cutting-edge AI, sophisticated control systems, and novel mechanical design to create adaptable, intelligent automation solutions capable of operating safely alongside humans in dynamic, real-world environments. At Amazon, we leverage the power of machine learning, artificial intelligence, and advanced robotics to solve some of the most complex operational challenges at a scale unlike anywhere else in the world. Our fleet of robots spans hundreds of facilities globally, working in sophisticated coordination to deliver on our promise of customer excellence — and we're just getting started. As a Sr. Scientist in Robot Navigation, you will be at the forefront of this transformation — architecting and delivering navigation systems that are intelligent, safe, and scalable. You will bring deep expertise in learning-based planning and control, a strong understanding of foundation models and their application to embodied agents, and as well as have in-depth understanding of control-theoretic approaches such as model predictive control (MPC)-based trajectory planning. You will develop navigation solutions that seamlessly blend data-driven intelligence with principled control-theoretic guarantees. Our vision is bold: to build navigation systems that allow robots to move fluidly and safely through dynamic environments — understanding context, anticipating change, and adapting in real time. You will lead research that bridges the gap between cutting-edge academic advances and production grade deployment, collaborating with world-class teams pushing the boundaries of robotic autonomy, manipulation, and human-robot interaction. Join us in building the next generation of intelligent navigation systems that will define the future of autonomous robotics at scale. Key job responsibilities - Design, develop, and deploy perception algorithms for robotics systems, including object detection, segmentation, tracking, depth estimation, and scene understanding - Lead research initiatives in computer vision, sensor fusion and 3D perception - Collaborate with cross-functional teams including robotics engineers, software engineers, and product managers to define and deliver perception capabilities - Drive end-to-end ownership of ML models — from data collection and labeling strategy to training, evaluation, and deployment - Mentor junior scientists and engineers; contribute to a culture of technical excellence - Define and track key metrics to measure perception system performance in real-world environments - Publish research findings in top-tier venues (CVPR, ICCV, ECCV, ICRA, NeurIPS, etc.) and contribute to patents A day in the life - Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment - Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations - Actively contribute to brainstorming sessions on adjacent topics, bringing fresh perspectives that help peers grow and succeed — and in doing so, build lasting trust across the team - Mentor team members while maintaining significant hands-on contribution to technical solutions About the team Our team is a group is a diverse group of scientists and engineers passionate about building intelligent machines. We value curiosity, rigor, and a bias for action. We believe in learning from failure and iterating quickly toward solutions that matter.
Economist II, GMAC Economics
GB, London
Are you excited about using econometrics, experimentation, and machine learning to impact real-world business decisions? We are looking for an Economist II to work on challenging problems at the intersection of causal inference and machine learning for Prime Video Ads. You will design experiments, build econometric and ML models, and translate findings into decisions that shape how millions of customers experience advertising on Prime Video. If you have a deeply quantitative approach to problem-solving, enjoy building and implementing models end-to-end, and want to work on problems where rigorous economics meets production-scale ML, we want to talk to you. Key job responsibilities - Design, execute, and analyze experiments to measure the impact of ad policies on customer behavior and business outcomes - Develop causal inference models (experimental and observational) to estimate short- and long-term effects of strategic initiatives - Collaborate with scientists, engineers, and product teams to deliver measurable business impact - Influence business leaders based on empirical findings
Sr. Applied Scientist, AWS Automated Reasoning
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities The successful candidate will: - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues. A day in the life This is a unique and rare opportunity to get in early on a fast-growing segment of AWS and help shape the technology, product and the business. You will have a chance to utilize your deep technical experience within a fast moving, start-up environment and make a large business and customer impact. About the team Diverse Experiences Amazon Automated Reasoning values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying. Why Amazon Automated Reasoning? At Amazon, automated reasoning is central to maintaining customer trust and delivering delightful customer experiences. Our organization is responsible for creating and maintaining a high bar for automated reasoning across all of Amazon's products and services. We offer talented automated reasoning professionals the chance to accelerate their careers with opportunities to build experience in a wide variety of areas including cloud, devices, retail, entertainment, healthcare, operations, and physical stores. Inclusive Team Culture In Amazon Automated Reasoning, it's in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest automated reasoning challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. Training & Career Growth We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve.
Applied Scientist, AWS Automated Reasoning
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities The successful candidate will: - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues. A day in the life This is a unique and rare opportunity to get in early on a fast-growing segment of AWS and help shape the technology, product and the business. You will have a chance to utilize your deep technical experience within a fast moving, start-up environment and make a large business and customer impact. About the team Diverse Experiences Amazon Automated Reasoning values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying. Why Amazon Automated Reasoning? At Amazon, automated reasoning is central to maintaining customer trust and delivering delightful customer experiences. Our organization is responsible for creating and maintaining a high bar for automated reasoning across all of Amazon's products and services. We offer talented automated reasoning professionals the chance to accelerate their careers with opportunities to build experience in a wide variety of areas including cloud, devices, retail, entertainment, healthcare, operations, and physical stores. Inclusive Team Culture In Amazon Automated Reasoning, it's in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest automated reasoning challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. Training & Career Growth We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve.
Applied Scientist II, Sponsored Products and Brands-Agent
US, NY, New York
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. About the team SPB Agent team's vision is to build a highly personalized and context-aware agentic advertiser guidance system that seamlessly integrates Large Language Models (LLMs) with sophisticated tooling, operating across all experiences. The SPB-Agent is the central agent that interfaces with advertisers across Ads Console, Selling Partner portals (Seller Central, KDP, Vendor Central), and internal Sales systems. We identify high-impact opportunities spanning from strategic product guidance to granular optimization and deliver them through personalized, scalable experiences grounded in state-of-the-art agent architectures, reasoning frameworks, sophisticated tool integration, and model customization approaches including fine-tuning, MCP, and preference optimization. This presents an exceptional opportunity to shape the future of e-commerce advertising through advanced AI technology at unprecedented scale, creating solutions that directly impact millions of advertisers.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social