Amazon at CVPR: Pietro Perona on computer vision's frontiers

Efficient learning and the capacity for abstraction are attributes that will probably require new insights — but self-supervised learning could help.

The Conference on Computer Vision and Pattern Recognition (CVPR) — the premier conference in the field of computer vision — was first held in 1985. Pietro Perona, an Amazon Fellow and the Allan E. Puckett Professor of Electrical Engineering and Computation and Neural Systems at the California Institute of Technology, first attended in 1988, when he was a graduate student at the University of California, Berkeley.

Pietro Perona.jpg
Pietro Perona, an Amazon Fellow and the Allan E. Puckett Professor of Electrical Engineering and Computation and Neural Systems at the California Institute of Technology.

“At the time, computer vision was a field for visionaries — pun intended — where we wanted to solve the question of how we can make a machine see,” Perona says. “The whole conference was maybe 200 people. And we had basically no clear idea how to make progress and so would try different things, and we would try and see if we could split the complex problem of vision into simpler questions. And the results were not very good. Now we see in the conference great systems working really well on very difficult problems. So the level of success and ambition is completely different.”

Much of that success stems, of course, from deep learning, which superseded many earlier computer vision techniques. But, Perona points out, it’s not as if computer vision researchers had simply failed to recognize the utility of deep learning for CVPR’s first 25 years. Until around 2010, he says, using deep learning to tackle computer vision problems wasn’t really an option.

“Deep learning has been around since the late ’80s,” he says, “but we simply didn't have enough computational power to run big experiments on complex images. You have to look to 2008, 2009, when good GPUs began coming out. Then, people in computer vision had to learn how to code up these GPUs. There were no special software tools at the time, so people were just handcrafting software.

“Another factor is the emergence of vast, well-annotated datasets of images, which came about in 2005 to 2010. That was the result of a couple of things. One was the Internet: all of a sudden there were tons of images available. The other thing is Amazon Mechanical Turk, which came out in 2005, and without which we would not be able to have these very large annotated datasets. It's funny, because within Amazon, people are not so aware of it, but Amazon Mechanical Turk was one of the three big factors for the AI revolution to come about. Datasets like ImageNet and COCO would not have been possible without it.”

Unscaled heights

For all of deep learning’s successes on such canonical computer vision tasks as object recognition, there are some respects in which it has made little headway, Perona says.

More on Amazon at CVPR

Read more about Amazon's presence at CVPR, including papers, workshop involvement, and committee membership.

“One barrier is the efficiency of learning,” he says. “There was a paper from my team looking at classification of plants and animals. If you have 10,000 images per category — each species of bird or species of butterfly — then the machine will beat a human in accuracy. But the efficiency is not even close. If I give you a new species you have never seen before, and I show you three to five pictures of this new species, you become competent at recognizing that species. For a machine that would not be possible.” 

One reason to try to break this barrier is scientific, Perona says. “Humans don't own a special kind of computation,” he says. “So it should be possible for machines to do it. You want to understand this exquisite ability that humans have, how it works.”

But, he adds, there are also practical reasons to worry about learning efficiency.

“If you think of people who are trying to use machine vision in industry or in science, something that is frequent is often not so important,” Perona explains. “What is rare is more important. So if you think of building a machine that can help an ophthalmologist recognize retinal disease, let's suppose, there are some 10 or 20 diseases of the retina that doctors see all the time. So they have no problem. They don't need help from a machine. But then there are another about 600 diseases that they see fewer times. And some of those are seen just by a few doctors per year. 

It's funny, because within Amazon, people are not so aware of it, but Amazon Mechanical Turk was one of the three big factors for the AI revolution to come about.
Pietro Perona

“The world is a long-tailed distribution. A few things are very frequent, and most things are not frequent at all. How often do you see an elephant cross the road? But if you want to build autonomous vehicles, they should be able to handle elephants crossing the road.”

Another aspect of human visual reasoning that deep learning has struggled to duplicate is the capacity for abstraction, Perona says.

“Right now, we need to train machines with diverse backgrounds,” he says. “If you want to train a machine to recognize toads, you've got to show it pictures of toads in all possible environments and all possible poses for the machine to be able to abstract away the concept of toad. If you had trained the machine with pictures of toads always against the same piece of wallpaper or the same blank background, the machine would not be able to handle the toad in a new scenario. Or take a cow on the beach: machines have a terrible time recognizing a cow that is right in the middle of a picture, and it's on the beach. So we know that machines are not yet seeing objects the same way we see them. From the training examples, they are not able to abstract away the attributes of these objects. What is the face of the cow? And relating the face of the cow with the face of a dog and the face of a person — the machine is not yet able to do that.”

Self-supervised learning

Before machines’ learning efficiency and capacity for abstraction can rival humans’, Perona says, “new insights are needed”. But in the near term, progress on both fronts could come from self-supervised learning, a topic that has, he says, grown in popularity at CVPR in recent years.

“Even if there is nobody teaching a machine what to look for, the machine can teach itself in some way and can be prepared to learn the next task,” Perona explains. “Let's suppose that we have a million images, for example, but no labels telling the machine what is in each picture. The machine has CPU cycles to spare, so what could it do? The images are all upside up, with the sky up and the ground down. But the machine could randomly flip a few and train itself to recognize when the image is flipped versus when the image is as it should be. Here’s another game you can play: each image is color, so there are three channels, RGB [red, green, blue]. So you could try and predict the green from the red and blue.

“Now, it turns out that in order to win at these games, it will have to develop some sense for the key features in the image. And one crucial feature is that trees grow from the ground up in some way. And so it has to recognize the structure of trees or the structure of things that are planted in the ground to recognize what is on the ground and what is not. It doesn't have a high level of semantic knowledge, but it does develop some features that are good preparation for the next step.

“To give you more advanced example, a student of mine and I have a paper showing how a machine can learn about numbers purely by playing with objects. Suppose that you had a few M&Ms, and you are just tossing them into a cup in front of you, and then you're picking one up and moving it away or putting one in or just scrambling the ones you have and rearranging them like a child would do. We demonstrate that the machine is able to learn the concept of number, an abstract concept, purely by playing with little objects, taking one out and putting one in, and so on. And it's quite interesting how that concept, that abstraction, can emerge from no supervision at all.”

Research areas
About the Author
Larry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.

Related content

US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for an Economist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.Economists at Amazon are solving some of the most challenging applied economics questions in the tech sector. Amazon economists apply the frontier of economic thinking to market design, pricing, forecasting, program evaluation, online advertising and other areas. Our economists build econometric models using our world class data systems, and apply economic theory to solve business problems in a fast-moving environment. A career at Amazon affords economists the opportunity to work with data of unparalleled quality, apply rigorous applied econometric approaches, and work with some of the most talented applied econometricians in the trade.As the Economist within WW Installments, you will be responsible for building long-term causal inference models and experiments. These analysis represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how objective functions are designed and which inputs are consumed for modeling. You will work across functions including machine learning, business intelligence, data engineering, software development, and finance to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing a causal inference and experimentation roadmap for the WW Installments Competitive Pricing team.• Apply expertise in causal and econometric modeling to develop large-scale systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the project plan from a scientific perspective on product launches including identifying potential risks, key milestones, and paths to mitigate risks• Own key inputs to reports consumed by VPs and Directors across Amazon.• Identifying new opportunities to influence business strategy and product vision using causal inference.• Continually improve the WW Installments experimentation roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver analytical projects and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for an Applied Scientist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.As an Applied Scientist within WW Installments, you will be responsible for building machine learning models and pipelines with direct customer impact. These models represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how they interact with financing options to make purchases. You will work across functions including data engineering, software development, and business to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing production machine learning models and pipelines for the WW Installments Competitive Pricing team that directly impact customers.• Apply expertise in machine learning to develop large-scale production systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the implementation of production ML from a scientific perspective including identifying potential risks, key milestones, and paths to mitigate risks.• Identifying new opportunities to influence business strategy and product vision using data science and machine learning.• Continually improve the WW Installments ML roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver ML pipelines, analytics projects, and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for a Data Scientist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.As a Data Scientist within WW Installments, you will be responsible for building machine learning models and pipelines with direct customer impact. These models represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how they interact with financing options to make purchases. You will work across functions including data engineering, software development, and business to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing machine learning models and pipelines for the WW Installments Competitive Pricing team.• Apply expertise in machine learning to develop large-scale systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the project plan from a scientific perspective on product launches including identifying potential risks, key milestones, and paths to mitigate risks.• Own key inputs to reports consumed by VPs and Directors across Amazon.• Identifying new opportunities to influence business strategy and product vision using data science and machine learning.• Continually improve the WW Installments ML roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver ML pipelines, analytics projects, and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, CA, San Diego
Job summaryPrivate Brands is fast-growing within Amazon, and is a highly visible, emerging business. We have a unique business and obsess over quality and building global brands our customers love. We aspire to be part of our customers’ everyday lives by offering them unique products at compelling prices backed by Amazon’s strong customer obsessed reputation.Private Brands Intelligence (PBI) is looking for a Data Scientist to join our team in building Machine Learning solutions at scale. PBI applies Machine Learning, Causal Inference, and Econometrics/Economics to derive actionable insights about the complex economy of Amazon’s retail business. We also develop statistical models and algorithms to drive strategic business decisions and improve operations. We are an interdisciplinary team of Economists, Scientists, and Engineers incubating and building Day One solutions using cutting-edge technology, to solve some of the toughest business problems at Amazon.You will work with business leaders, scientists, economists, and engineers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed services. You will partner with scientists, economists, and engineers to help invent and implement scalable ML and econometric models while building tools to help our customers gain and apply insights.This is a unique, high visibility opportunity for someone who wants to have business impact, dive deep into large-scale economic problems, enable measurable actions on the Consumer economy, and work closely with scientists and economists. We are particularly interested in candidates with experience building predictive models and working with distributed systems.As a Data Scientist, you bring business and industry context to science and technology decisions. You set the standard for scientific excellence and make decisions that affect the way we build and integrate algorithms. Your solutions are exemplary in terms of algorithm design, clarity, model structure, efficiency, and extensibility. You tackle intrinsically hard problems, acquiring expertise as needed. You decompose complex problems into straightforward solutions.
US, VA, Arlington
Job summaryThis role will sit in our new headquarters in Northern Virginia, where Amazon will invest $2.5 billion dollars, occupy 4 million square feet of energy efficient office space, and create at least 25,000 new full-time jobs.The AWS Infrastructure Data Center Planning and Delivery (DCPD) Data Science team owns supply chain management activities at a global scale.We consolidate usage and supply chain health data and forecasts at a variety of horizons to ensure that we have the right strategic lens associated with each decision we make.We identify gaps to ensure that the AWS business is able to support any and all customers who want to capitalize on the scalability, flexibility, and cost-efficiency of AWS. Our actions and decisions decide the where, how, and what will make it into each of our data centers and we need you to help us to make those decisions and clearly explain the why.The Business Insights and Optimization (BIO) team owns data science, engineering, and business intelligence solutions feeding this team.We identify gaps in our capacity planning and delivery mechanisms and design/build systems which will fix those gaps.We are end to end data product owners and the analysis, models we produce drives billions of dollars of decisions annually.Data Scientists on this team have end to end range and capabilities.They work directly with business owners to understand how they use data to drive their business.They design modeling frameworks to dive deep into these raw sources of information to get the most out of the data they have.They work directly with data engineers to build automated pipelines and production scale information systems and models.They build automated tools which will allow their results to be shared with the business at scale.They align with business owners to continuously track their work to ensure maximum impact from their projects.They monitor performance of their work to evaluate whether improvements are needed after tracking has started in production.
US, CA, Sunnyvale
Job summaryAmong the goals of the Alexa Devices AI team, is to make Alexa the most knowledgeable and trusted ally for notifications, annoucements, pickup services and voice assistance while on the go.Key job responsibilities1. As an Applied Scientist on our team you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art NLU (Natural language understanding) developments.2. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to traing Machine Learning models for their application in NLU.3. This role requires a pragmatic technical leader comfortable with ambiguity, capable of summarizing complex data and models through clear visual and written explanations.4. The ideal candidate will have experience with machine learning models and their application in AI systems. We are particularly interested in experience applying natural language processing, deep learning at scale. Additionally, we are seeking candidates with strong interest in data/research sciences and engineering, creativity, curiosity, and great judgment.5. You will interact with various stake holders: product leaders, program managers, other domain managers and developers on regular basis for requirement collections, deliveries, and other related communication6. You will help attract and recruit technical talentA day in the lifeApplied Scientist will help develop novel algorithms and apply modeling techniques to advance the state of the art in spoken language understanding (SLU) and to improve the customer experience in engaging with Alexa.About the teamThe Alexa Devices AI science team's work directly impacts the experience and engagement of customers who rely on Alexa while in-the-car, on-the-go and at-home.
US, VA, Arlington
Job summaryThe Central Science Team within Amazon’s People Experience and Technology org (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal.We are looking for an economist with expertise in applying causal inference, experimental design, or causal machine learning techniques to topics in labor, personnel, education, health, public, or behavioral economics. We are particularly interested in candidates with experience applying these skills to strategic problems with significant business and/or social policy impact.Candidates will work with economists and engineers to estimate and validate their models on large scale data, and will help business partners turn the results of their analysis into policies, programs, and actions that have a major impact on Amazon’s business and its workforce. We are looking for creative thinkers who can combine a strong economic toolbox with a desire to learn from others, and who know how to execute and deliver on big ideas.Ideal candidates will own key inputs to all stages of research projects, including model development, survey administration, experimental design, and data analysis. They will be customer-centric, working closely with business partners to define key research questions, communicate scientific approaches and findings, listen to and incorporate partner feedback, and deliver successful solutions.
US, CA, Palo Alto
Job summaryAmazon is the 4th most popular site in the US (http://www.alexa.com/topsites/countries/US). Our product search engine is one of the most heavily used services in the world, indexes billions of products, and serves hundreds of millions of customers world-wide. We are working on a new AI-first initiative to re-architect and reinvent the way we do search through the use of extremely large scale next-generation deep learning techniques. Our goal is to make step function improvements in the use of advanced Machine Learning (ML) on very large scale datasets, specifically through the use of aggressive systems engineering and hardware accelerators. This is a rare opportunity to develop cutting edge ML solutions and apply them to a problem of this magnitude. Some exciting questions that we expect to answer over the next few years include:· Can a focus on compilers and custom hardware help us accelerate model training and reduce hardware costs?· Can combining supervised multi-task training with unsupervised training help us to improve model accuracy?· Can we transfer our knowledge of the customer to every language and every locale ? The Search Science team is looking for a Senior Applied Science Manager to drive roadmap on making large business impact through application of Deep Learning models via close collaboration with partner teams. The team also has a focus on technology solution for deep-learning based embedding generation, sensitive data ingestion and applications, data quality measurement, improvement, data bias identification and reduction to achieve model fairness.Success in this role will require the courage to chart a new course. You will manage your own team to understand all aspects of the customer journey. You and your team will inform other scientists and engineers by providing insights and building models to help improving training data quality and reducing bias. The research focus includes but not limited to Natural Language Processing, recommendation, applications relevant to Amazon buyers, sellers and more. You will be working with cutting edge technologies that enable big data and parallelizable algorithms. You will play an active role in translating business and functional requirements into concrete deliverables and working closely with software development teams to put solutions into production.
US, WA, Seattle
Job summaryAmazon EC2 provides cloud computing which forms the foundation for the majority of AWS services, as well as a large portion of compute use cases for businesses and individuals around the world. A critical factor in the continued success of EC2 is the ability to provide reliable and cost effective computing. The EC2 Fleet Health and Lifecycle (EC2 FHL) organization is responsible for ensuring that the global EC2 server fleet continues to raise the bar for reliability, security, and efficiency. We are looking for seasoned engineering leaders with passion for technology and an entrepreneurial mindset. At Amazon, it is all about working hard, having fun and making history. If you are ready to make history, we want to hear from you!Come join a brand new team, EC2 Health Analytics, under EC2 Foundational Technology, to solve complex cutting-edge problems to power a faster, more robust and performant EC2 of tomorrow. The charter of our team is to improve customer experience on the EC2 fleet by analyzing hundreds of signals and driving next-generation detection and remediation tools. We apply Machine Learning to predict outcomes and optimize decisions that improve customer experience and operational efficiency. As an Applied Scientist in the EC2 Health Analytics team, you will join an industry-leading engineering team solving challenging problems at massive scale.· Build a world-class forecasting platform that scales to handling billions of time series data in real time.· Drive fleet utilization improvement where each 1% means tens of millions of additional free cash flow.· Automate tactical and strategic capacity planning tools to optimize for service availability and infrastructure cost.· Build recommendation algorithms for improving the AWS customer experience.· · Reduce dependence on manual troubleshooting for deep-dives.What you will learn:· State-of-the-art analytics and forecasting methodologies.· Application of machine learning to large-scale data sets.· · Product recommendation algorithms.· Resource management and admission control for the Cloud.· The internals of all AWS services.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, CA, Palo Alto
Job summaryThe Amazon Search team creates powerful, customer-focused search and advertising solutions and technologies. Whenever a customer visits an Amazon site worldwide and types in a query or browses through product categories, the Amazon Search services go to work. We design, develop, and deploy high performance, fault-tolerant distributed search systems used by millions of Amazon customers every day. Our team works to maximize the quality and effectiveness of the search experience for visitors to Amazon websites worldwide.