Amazon takes top three spots in Audio Anomaly Detection Challenge

Team from Amazon Web Services also wins the best-paper award at the Workshop on Detection and Classification of Acoustic Scenes and Events.

This week at Amazon Web Services’ re:Invent 2020 conference, Amazon announced Amazon Monitron, an end-to-end machine-monitoring system composed of sensors, a gateway, and a machine learning model that detects anomalies in vibration (structure-borne sound) or temperature and predicts when equipment may require maintenance. 

Machine condition monitoring was also the topic of a challenge at the Workshop on the Detection and Classification of Acoustic Scenes and Events (DCASE 2020), in November, in which Amazon took the top three spots, out of 117 submissions.

The challenge was to determine whether the sounds emitted by a machine — such as a fan, pump, or valve — were normal or anomalous. Forty academic and industry teams submitted entries, an average of almost three submissions per team.

In a pair of papers (paper 1|paper 2) we presented at the workshop, we describe the two different neural-network-based approaches we took in our submissions to the challenge. The first of those papers won the workshop’s best-paper award.

Spectrograms of audio clips recorded from a normal valve and a faulty valve.
Spectrograms of audio clips recorded from a normal valve (top) and a faulty valve (bottom). The magnified details show the difference between the normal signal and the anomalous signal.

Auditory machine condition monitoring has been common in industrial settings for several decades. Seasoned maintenance experts can identify problems in the machines they monitor just by listening to them and realizing that “something doesn’t sound right.” But by the time anomalies are audible to the human ear, the underlying problems may already be well advanced.

With the advent of machine learning and big data, there has been a lot of interest in teaching machines to detect anomalies sooner, to help predict when preventative maintenance might be necessary.

Data, labels, and rare failures

In general, anomaly detection is the problem of identifying abnormal inputs in a stream of inputs. Depending on the available data, there are three different ways to train anomaly detection systems: (i) fully supervised, in which labeled examples of normal and abnormal data are presented; (ii) semi-supervised, in which only normal data is presented; and (iii) unsupervised, in which there are no labels in the data set, and outliers have to be classified automatically. 

Anomalies can manifest themselves in different ways. For instance, you can have slow concept drift or sudden, instantaneous outliers. Typically, the data is also highly imbalanced — a lot more “normal” examples than “abnormal.”

Machines worth monitoring carefully — especially those that are critical or expensive — are usually also well maintained. This means that they rarely fail, and gathering anomalous data from them is challenging and may take many years and lots of effort.

Additionally, machines operate in different modes and under variable load or performance conditions, and their characteristics can change over time as they age and approach steady state. Some industries’ operational profiles have seasonal variations as well. 

All of these factors make anomaly detection challenging in the industrial setting. When implementing an anomaly detection system, one has to depend mostly on “normal” data, gathering additional data over time and eliciting user feedback. 

If accurate physical models of machines are available, it may be possible to simulate failures and generate “abnormal” data that way. One can also generate anomalous data by inducing hardware failures in the lab. But one has to be prepared to work with minimal data when a machine is instrumented for the first time (the so-called cold-start problem).

Anomaly detection and our two neural approaches

The papers we presented at DCASE (paper1|paper2) describe two different neural-network-based approaches to anomaly detection.

The first approach builds on recent advances in autoregressive neural-density estimation, or calculating a data distribution for streaming data by trying to predict each new data item on the basis of those that preceded it. As might be expected, such models are very sensitive to the order in which data arrives.

An earlier model, called the masked autoencoder for density estimation (MADE), makes a separate prediction for each feature — each dimension — of the input. With audio signals, however, the dimensions of the input are the energies in different frequency bands, which produce a composite picture of the signal that individual frequencies won’t capture. 

We introduce a variation of MADE that bases its predictions on groups of input features — in this case, groups of frequency bands — and which we accordingly call Group MADE.

In the second paper, we use a self-supervised approach for representation learning, which has been successful recently in solving problems in vision and speech. We believe that we are the first to apply it to audio anomaly detection. 

In the absence of anomalies in the training data, we trained a network to instead learn to distinguish multiple instances of machines within a given machine type. We found that the features learned by such a network were sensitive enough to detect delicate, previously unseen anomalies in the evaluation set. We used spectral warping and random mixing to simulate new machine instances in addition to the ones provided in the dataset. 

Two-dimensional visualizations of two different representations of the Toy Car sounds in the DCASE data set.
Two-dimensional visualizations of two different representations of the Toy Car sounds in the DCASE data set: the raw spectrograms (left) and the features learned by our self-supervised model (right). The blue samples represent data from a normal machine, the red samples data from an anomalous one. In the raw spectrograms, there is little separation between the normal and anomalous samples, while the learned features can much more clearly separate out the anomalous samples.

The DCASE challenge provided data from six different machines: fan, pump, slide rail, valve, toy car, and toy conveyor. DCASE also provided a development data set and a separate evaluation data set. Scoring was calculated using area under the ROC curve (AUC) and partial area under the ROC curve. The ROC curve maps false-positive rate against false-negative rate, so the area under the curve indicates how well a given system manages that trade-off; partial AUC is the AUC over a small false-positive-rate range, in this case [0, 0.1]. 

The table below shows the accuracies we were able to obtain, both for the challenge and since the challenge. We have developed a third approach that helped improve some of these numbers, which we will detail in a future publication. 

The challenge ranking method involved two steps, to account for the the disparate difficulty levels across various machine types. First, machine-specific rankings were assigned to all submissions, based on AUC and pAUC. The submissions were then ranked by the average of their machine-specific ranks. Please see the full leaderboard here.

While our models won the challenge using the across-all-machine-types scoring described above, fine-tuning them for specific machine types yielded the results in the last row.

DCASE results table

We believe that as more industrial machine data is accumulated and curated over the next few years, machine learning and neural-network-based approaches will start making a huge difference in the monitoring and maintenance of machines, and AWS and its services will be at the forefront of this revolution.

Research areas

Related content

US, MA, North Reading
We are looking for experienced scientists and engineers to explore new ideas, invent new approaches, and develop new solutions in the areas of Controls, Dynamic modeling and System identification. Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Key job responsibilities Applied Scientists take on big unanswered questions and guide development team to state-of-the-art solutions. We want to hear from you if you have deep industry experience in the Mechatronics domain and : * the ability to think big and conceive of new ideas and novel solutions; * the insight to correctly identify those worth exploring; * the hands-on skills to quickly develop proofs-of-concept; * the rigor to conduct careful experimental evaluations; * the discipline to fast-fail when data refutes theory; * and the fortitude to continue exploring until your solution is found We are open to hiring candidates to work out of one of the following locations: North Reading, MA, USA | Westborough, MA, USA
GB, London
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python or R is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: London, GBR
IN, KA, Bengaluru
Job Description ATE (Analytics, Technology and Engineering) is a multi-disciplinary team of scientists, engineers, and technicians, all working to innovate in operations for the benefit of our customers. Our team is responsible for creating core analytics, science capabilities, platforms development and data engineering. We develop scalable analytics applications and research modeling to optimize operation processes.. You will work with professional software development managers, data engineers, data scientists, applied scientists, business intelligence engineers and product managers using rigorous quantitative approaches to ensure high quality data tech products for our customers around the world, including India, Australia, Brazil, Mexico, Singapore and Middle East. We are on the lookout for an enthusiastic and highly analytical individual to be a part of our journey. Amazon is growing rapidly and because we are driven by faster delivery to customers, a more efficient supply chain network, and lower cost of operations, our main focus is in the development of strategic models and automation tools fed by our massive amounts of available data. You will be responsible for building these models/tools that improve the economics of Amazon’s worldwide fulfillment networks in emerging countries as Amazon increases the speed and decreases the cost to deliver products to customers. You will identify and evaluate opportunities to reduce variable costs by improving fulfillment center processes, transportation operations and scheduling, and the execution to operational plans. You will also improve the efficiency of capital investment by helping the fulfillment centers to improve storage utilization and the effective use of automation. Finally, you will help create the metrics to quantify improvements to the fulfillment costs (e.g., transportation and labor costs) resulting from the application of these optimization models and tools. Major responsibilities include: · In this role, you will be responsible for developing and implementing innovative, scalable models and tools aimed at tackling novel challenges within Amazon’s global fulfillment network. Collaborating with fellow scientists from various teams, you will work on integrated solutions to enhance fulfillment speed, reduce costs. Your in-depth comprehension of business challenges will enable you to provide scientific analyses that underpin critical business decisions, utilizing a diverse range of methodologies. You’ll have the opportunity to design scientific tool platforms, deploy models, create efficient data pipelines, and streamline existing processes. Join us in shaping the future of Amazon’s global retail business by optimizing delivery speed at scale and making a lasting impact on the world of e-commerce. If you’re passionate about solving complex problems and driving innovation, we encourage you to apply. About the team This team is responsible for applying science based algo and techniques to solve the problems in operation and supply chain. Some of these problems include, volume forecasting, capacity planning, fraud detection, scenario simulation and using LLM/GenAI for process efficiency We are open to hiring candidates to work out of one of the following locations: Bengaluru, KA, IND
IL, Tel Aviv
Are you passionate about pushing the boundaries of computer vision, generative AI, deep learning, and machine learning? Ready to tackle challenges in document understanding at scale? We’re looking for innovative minds to join our world-class team at AWS, where you’ll collaborate with leading researchers, academics, and engineers on Amazon Textract. Why AWS? Be part of the leading cloud service provider powering innovation and positive impact. Work on real-world problems alongside tech and business giants. Access to unlimited data and computational resources. Collaborate with world-class researchers and developers. Deploy solutions at AWS scale and publish your work at top conferences. Focus Areas: - LLMs, document understanding, scene text recognition. - Visual question answering, NLP+vision, layout understanding. Locations: Tel Aviv and Haifa Think you’re a fit? Dive into the world of AWS Computer Vision and help us innovate at the forefront of technology. Key job responsibilities - Design cutting-edge neural network architectures. - Create document understanding solutions for complex scenarios and large visual datasets. - Set benchmarks and success criteria for model performance. - Collaborate across AWS and Amazon to bring scientific breakthroughs to our customers. - Add your unique creativity to our multidisciplinary team. - Mentor junior scientists and interns/PhD students. We are open to hiring candidates to work out of one of the following locations: Haifa, ISR | Tel Aviv, ISR
LU, Luxembourg
Have you ever wished to build high standard Operations Research and Machine Learning algorithms to optimize one of the most complex logistics network? Have you ever ordered a product on Amazon websites and wondered how it got delivered to you so fast, and what kinds of algorithms & processes are running behind the scenes to power the whole operation? If so, this role is for you. The team: Global transportation services, Research and applied science - Operations is at the heart of the Amazon customer experience. Each action we undertake is on behalf of our customers, as surpassing their expectations is our passion. We improve customer experience through continuously optimizing the complex movements of goods from vendors to customers throughout Europe. - Global transportation analytical teams are transversal centers of expertise, composed of engineers, analysts, scientists, technical program managers and developers. We are focused on Amazon most complex problems, processes and decisions. We work with fulfillment centers, transportation, software developers, finance and retail teams across the world, to improve our logistic infrastructure and algorithms. - GTS RAS is one of those Global transportation scientific team. We are obsessed by delivering state of the art OR and ML tools to support the rethinking of our advanced end-to-end supply chain. Our overall mission is simple: we want to implement the best logistics network, so Amazon can be the place where our customers can be delivered the next-day. The role: Applied scientist, speed and long term network design The person in this role will have end-to-end ownership on augmenting RAS Operation Research and Machine Learning modeling tools. They will help understand where are the constraints in our transportation network, and how we can remove them to make faster deliveries at a lower cost. You will be responsible for designing and implementing state-of-the-art algorithmic in transportation planning and network design, to expand the scope of our Operations Research and Machine Learning tools, to reflect the constantly evolving constraints in our network. You will enable the creation of a product that drives ever-greater automation, scalability and optimization of every aspect of transportation, planning the best network and modeling the constraints that prevent us from offering more speed to our customer, to maximize the utilization of the associated resources. The impact of your work will be in the Amazon EU global network. The product you will build will span across multiple organizations that play a role in Amazon’s operations and transportation and the shopping experience we deliver to customer. Those stakeholders include fulfilment operations and transportation teams; scientists and developers, and product managers. You will understand those teams constraints, to include them in your product; you will discuss with technical teams across the organization to understand the existing tools and assess the opportunity to integrate them in your product.You will engage with fellow scientists across the globe, to discuss the solutions they have implemented and share your peculiar expertise with them. This is a critical role and will require an aptitude for independent initiative and the ability to drive innovation in transportation planning and network design. Successful candidates should be able to design and implement high quality algorithm solutions, using state-of-the art Operations Research and Machine Learning techniques. Key job responsibilities - Engage with stakeholders to understand what prevents them to build a better transportation network for Amazon - Review literature to identify similar problems, or new solving techniques - Build the mathematical model representing your problem - Implement light version of the model, to gather early feed-back from your stakeholders and fellow scientists - Implement the final product, leveraging the highest development standards - Share your work in internal and external conferences - Train on the newest techniques available in your field, to ensure the team stays at the highest bar About the team GTS Research and Applied Science is a team of scientists and engineers whom mission is to build the best decision support tools for strategic decisions. We model and optimize Amazon end-to-end operations. The team is composed of enthusiastic members, that love to discuss any scientific problem, foster new ideas and think out of the box. We are eager to support each others and share our unique knowledge to our colleagues. We are open to hiring candidates to work out of one of the following locations: Luxembourg, LUX
US, CA, Santa Clara
Amazon AI is looking for world class scientists and engineers to join its AWS AI Labs. This group is entrusted with developing core data mining, natural language processing, deep learning, and machine learning algorithms for AWS. You will invent, implement, and deploy state of the art machine learning algorithms and systems. You will build prototypes and explore conceptually new solutions. You will interact closely with our customers and with the academic community. You will be at the heart of a growing and exciting focus area for AWS and work with other acclaimed engineers and world famous scientists. Inclusive Team Culture Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future. We are open to hiring candidates to work out of one of the following locations: New York, NY, USA | Santa Clara, CA, USA | Seattle, WA, USA
DE, BE, Berlin
Are you excited about developing state-of-the-art computer vision models that revolutionize Amazon’s Fulfillment network? Are you looking for opportunities to apply AI on real-world problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics, we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience — at Amazon scale. To this end, we are looking for an Applied Scientist who will build and deploy models that make smarter decisions on a wide array of multi-modal signals. Together, we will be pushing beyond the state of the art in optimizing one of the most complex systems in the world: Amazon's Fulfillment Network. Key job responsibilities In this role, you will build computer vision and multi-modal deep learning models that understand the state of products and packages flowing through Amazon’s fulfillment network. You will build models that solve challenging problems like product identification and damage detection on Amazon's entire retail catalog (billions of different items, thousands of new items every day). You will primarily work with very large real-world vision datasets, as well as a diverse set of multi-modal datasets, including natural language and structured data. You will face a high level of research ambiguity and problems that require creative, ambitious, and inventive solutions. A day in the life AFT AI delivers the AI solutions that empower Amazon’s fulfillment network to make smarter decisions. You will work on an interdisciplinary team of scientists and engineers with deep expertise in developing cutting-edge AI solutions at scale. You will work with images, videos, natural language, and sequences of events from existing or new hardware. You will adapt state-of-the-art machine learning and computer vision techniques to develop solutions for business problems in the Amazon Fulfillment Network. About the team Amazon Fulfillment Technologies (AFT) powers Amazon’s global fulfillment network. We invent and deliver software, hardware, and science solutions that orchestrate processes, robots, machines, and people. We harmonize the physical and virtual world so Amazon customers can get what they want, when they want it. AFT AI is spread across multiple locations in NA (Bellevue WA and Nashville, TN) and Europe (Berlin, Germany). We are hiring candidates to work out of the Berlin location. Publicly available articles showcasing some of our work: - Damage Detection: https://www.amazon.science/latest-news/the-surprisingly-subtle-challenge-of-automating-damage-detection - Product ID: https://www.amazon.science/latest-news/how-amazon-robotics-is-working-on-new-ways-to-eliminate-the-need-for-barcodes We are open to hiring candidates to work out of one of the following locations: Berlin, BE, DEU
US, WA, Seattle
Here at Amazon, we embrace our differences. We are committed to furthering our culture of diversity and inclusion of our teams within the organization. How do you get items to customers quickly, cost-effectively, and—most importantly—safely, in less than an hour? And how do you do it in a way that can scale? Our teams of hundreds of scientists, engineers, aerospace professionals, and futurists have been working hard to do just that! We are delivering to customers, and are excited for what’s to come. Check out more information about Prime Air on the About Amazon blog (https://www.aboutamazon.com/news/transportation/amazon-prime-air-delivery-drone-reveal-photos). If you are seeking an iterative environment where you can drive innovation, apply state-of-the-art technologies to solve real world delivery challenges, and provide benefits to customers, Prime Air is the place for you. Come work on the Amazon Prime Air Team! Prime Air is seeking an experienced Research Scientist in the Flight Sciences High-Fidelity Methods (HFM) team within Flight Sciences, you will develop and verify aerodynamics models used for engineering analyses and vehicle simulation. These models are the backbone of every flight simulation performed within Prime Air and are a critical element in the aircraft design, verification and certification process. These models are used to predict many attributes of the vehicle performance including range, maneuverability, tracking error, and aircraft stability. They are a key input to design decisions, vehicle component sizing and flight software algorithm development. The accuracy and reliability of these flight model are critical to the success of Prime Air. For this role we are looking for a scientist to develop surrogate or machine learning models to represent the complex aerodynamic behavior of our drones. This scientist will develop techniques to validate these models using flight testing, quantify the model uncertainty, and assess the impact of this uncertainty on downstream engineering analyses. Key job responsibilities A Research Scientist in this role is responsible for owning the development, deployment, verification, and maintenance of models from end-to-end. This includes the initial gathering of the downstream customer needs, identifying the most suitable modelling approach, coordinating the generation of input data, training models, developing and maintaining software interfaces, and verifying the model accuracy. A Research Scientist in this role is responsible for determining the most suitable modeling approach for a given physical phenomena. They need to possess knowledge of various machine learning techniques, and their respective advantages and limitations. They will need to have a detailed understanding of the types of physics to be modelled including vehicle aerodynamics, multibody dynamics, and atmosphere physics. This role is responsible for designing experiments for generating data used to train and verify surrogate models. They need to have a basic understanding of the methods used to generate high-fidelity aerodynamics predictions including CFD, wind tunnel testing, and flight testing. They will be responsible for validating the models by leveraging uncertainty quantification, system identification, and statical analyses. Export Control License This position may require a deemed export control license for compliance with applicable laws and regulations. Placement is contingent on Amazon’s ability to apply for and obtain an export control license on your behalf. A day in the life A Research Scientist in the High-Fidelity Methods (HFM) team will have the opportunity to work on a wide variety of tasks. The ideal candidate should be adaptable and thrive in an everchanging environment. Depending on the phase of model or vehicle development, a typical day might consist of reading research papers on machine learning techniques, developing test plans for wind tunnel testing, writing code to train and verify models, reviewing flight test results, or writing documentation. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. As a Applied Scientist at the intersection of machine learning and the life sciences, you will participate in developing exciting products for customers. Our team rewards curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the cutting edge of both academic and applied research in this product area, you have the opportunity to work together with a diverse and talented team of scientists, engineers, and product managers and collaborate with others teams. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Bellevue
As a Principal Research Scientist in the Amazon Artificial General Intelligence (AGI) Data Services organization, you will be responsible for sourcing and quality of massive datasets powering Amazon's AI. You will play a critical role in driving innovation and advancing the state-of-the-art in natural language processing and machine learning. You will be responsible for developing and implementing cutting-edge algorithms and techniques to extract valuable insights from large-scale data sources. You will work closely with cross-functional teams, including product managers, engineers, and data scientists to ensure that our AI systems are aligned with human policies and preferences. Key job responsibilities - Responsible for sourcing and quality of massive datasets powering Amazon's AI. - Collaborate with cross-functional teams to ensure that Amazon’s AI models are aligned with human preferences. - Develop and implement strategies to improve the efficiency and effectiveness of programs delivering massive datasets. - Identify and prioritize research opportunities that have the potential to significantly impact our AI systems. - Communicate research findings and progress to senior leadership and stakeholders. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA | Boston, MA, USA