Why Alexa won't wake up when she hears her name in Amazon's Super Bowl ad

This Sunday's Super Bowl between the New England Patriots and the Los Angeles Rams is expected to draw more than 100 million viewers, some of whom will have Alexa-enabled devices within range of their TV speakers. When Amazon's new Alexa ad airs, and Forest Whitaker asks his Alexa-enabled electric toothbrush to play his podcast, how will we prevent viewers’ devices from mistakenly waking up?

Related content
In its collaboration with the NFL, AWS contributes cloud computing technology, machine learning services, business intelligence services — and, sometimes, the expertise of its scientists.

With the Super Bowl ad — as with thousands of other media mentions of Alexa tracked by our team — we teach Alexa what individual recorded instances of her name sound like, so she will know to ignore them. We can also apply this technique, known as acoustic fingerprinting, on the fly to recognize when multiple devices from different households are hearing the same command at around the same time. This is crucial to preventing Alexa from responding to pranks on TV, references to people named Alexa, or other instances of her name in broadcast media that we don't know about in advance.

Related content
Audio watermarking is the process of adding a distinctive sound pattern — undetectable to the human ear — to an audio signal to make it identifiable to a computer. It’s one of the ways that video sites recognize copyrighted recordings that have been posted illegally. To identify a watermark, a computer usually converts a digital file into an audio signal, which it processes internally.

Our approach to matching audio recordings is based on classic acoustic-fingerprinting algorithms like that of Haitsma and Kalker in their 2002 paper “A Highly Robust Audio Fingerprinting System”. Such algorithms are designed to be robust to audio distortion and interference, such as those introduced by TV speakers, the home environment, and our microphones.

To produce an acoustic fingerprint, we first derive a grid of log filter-bank energies (LFBEs) for the acoustic signal, which represent the amounts of energy in multiple overlapping frequency bands in a series of overlapping time windows. The algorithm steps through the grid in two-by-two blocks and adds and subtracts the measurements in the grid cells in a standardized way. (Technically, it computes the 2-D gradient of each block.) The sign of the result — positive or negative — provides a one-bit summary of the values in the block. The summaries of all the blocks in the grid constitute the acoustic fingerprint, and two fingerprints are deemed to match if the fraction of bits that are different (the “bit error rate”) is small enough.

Acoustic-fingerprinting_figure.jpg._CB455311870_.jpg
An illustration of how fingerprints are used to match audio. Different instances of Alexa’s name result in a bit error rate of about 50% (random bit differences). A bit error rate significantly lower than 50% indicates two recordings of the same instance of Alexa’s name.

When we have audio samples in advance — as we do with the Super Bowl ad — we fingerprint the entire sample and store the result. With audio that’s streaming to the cloud from Alexa-enabled devices, we build up fingerprints piecemeal, repeatedly comparing them to other fingerprints as they grow.

If a match is found, the incoming request is ignored. Noisy audio may yield a match, but it requires the accumulation of more data (a larger fingerprint) than clean audio does.

Using this matching algorithm, we have built a system with multiple layers to protect customers at multiple stages:

  • On-device: On most Echo devices, every time the wake word “Alexa” is detected, the audio is checked against a small set of known instances where Alexa is mentioned in commercials. Due to the limits of device CPU, this set is generally restricted to commercials we expect to be currently airing on TV.
  • In the cloud: Every audio request to Alexa that starts with a wake word is checked in two ways:
    • Known media: the audio is checked against a large set of fingerprints for known instances of “Alexa” and other wake words in commercials and other media. These fingerprints can also make use of the audio that follows the wake word.
    • Unknown media: the audio is checked against a fraction of other Alexa requests arriving at around the same time. If the audio of a request matches that of requests from at least two other customers, we identify it as a media event. We also check incoming audio against a small cache of fingerprints discovered on the fly (the cached fingerprints are averages of the fingerprints that were declared matches). The cache allows Alexa to continue to ignore spurious wake words even when they no longer occur simultaneously.

Ideally, a device will identify media audio using locally stored fingerprints, so it does not wake up at all. If it does wake up, and we match the media event in the cloud, the device will quickly and quietly turn back off.

In addition to tracking new media mentions of Alexa’s name and updating our library of fingerprints accordingly, our team works continuously to improve the accuracy and efficiency of the fingerprinting system. We’re also exploring complementary technologies, such as machine learning systems that can distinguish media audio more generally from live human speech.

Acknowledgments: Joe Wang, Aaron Challenner, Mike Peterson, Michael Rudeen, Naresh Narayanan, Liangwei Guo, and the rest of the team

Related content

US, WA, Seattle
Are you a Ph.D. interested in the fields of machine learning, deep learning, automated reasoning, speech, robotics, computer vision, optimization, or quantum computing? Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way? If this describes you, come join our science teams at Amazon. As an Applied Scientist, you will have access to large datasets with billions of images and video to build large-scale systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept. We are looking for smart scientists capable of using a variety of domain expertise to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.
LU, Luxembourg
Have you ever wondered how Amazon delivers timely and reliably hundreds of millions of packages to customer’s doorsteps? Are you passionate about data and mathematics, and hope to impact the experience of millions of customers? Are you obsessed with designing simple algorithmic solutions to very challenging problems?If so, we look forward to hearing from you!Amazon Transportation Services is seeking a Postdoctoral Scientist with Operations Research or Applied Mathematics background, to join our team in the EU Headquarters in Luxembourg, for a one-plus-one year full-time research position. As a key member of the EU Research Science Team, this person will be responsible for designing and implementing beyond state of the art algorithmic frameworks that optimize the middle-mile Amazon Transportation Network. The successful applicant will ensure that our end-to-end strategies in terms of customer demand fulfillment, routing, consolidation locations, linehaul/airhaul/sea options and last-mile transportation are streamlined and optimizedKey job responsibilitiesIn this role you will:• Work closely with a senior science advisor, collaborate with other scientists and engineers, and be part of Amazon’s vibrant and diverse global science community.• Publish your innovation in top-tier academic venues and hone your presentation skills.• Be inspired by challenges and opportunities to invent cutting-edge techniques in your area(s) of expertise.
ES, B, Barcelona
Are you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon delivery network. Current research and areas of work within our team include machine learning forecast, anomaly detection models, model interpretability, graph neural nets, among others.We are looking for a Manager, Applied Science (Machine Learning) with a strong academic background and industry experience in the areas of probabilistic machine learning, time series forecasting, and/or anomaly detection.At Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
NL, Amsterdam
Are you a passionate scientist in the computer vision area who is aspired to apply your skills to bring value to millions of customers? Here at Ring, we have a unique possibility to innovate and see how the results of our work improve the lives of millions of people and make neighborhoods safer.You will be part of a team committed to pushing the frontier of computer vision and machine learning technology to deliver the best experience for our neighbors. This is a great opportunity for you to innovate in this space by developing highly optimized algorithms that will work on scale. This position requires experience with developing efficient computer vision algorithms on resource-constrained computing platforms on edge. You will collaborate with different Amazon teams to make informed decisions on the best practices in machine learning to build highly-optimized integrated hardware and software platforms.Key job responsibilities* Research and implement the state-of-the-art computer vision and sensor fusion algorithms for resource-constrained computing platforms on a large scale.* Collaborate with product managers and engineering teams to design and implement computer vision and machine learning based features for Ring devices* Influence system design and product vision by making informed decisions on the selection of technology, data sources, algorithms, and sensors.
US, WA, Seattle
Amazon internships are full-time (40 hours/week) for 12 consecutive weeks with start dates in May - July 2023. Our internship program provides hands-on learning and building experiences for students who are interested in a career in hardware engineering. This role will be based in Seattle, and candidates must be willing to work in-person.Corporate Projects (CPT) is a team that sits within the broader Corporate Development organization at Amazon. We seek to bring net-new, strategic projects to life by working together with customers and evolving projects from ZERO-to-ONE. To do so, we deploy our resources towards proofs-of-concept (POCs) and pilot programs and develop them from high-level ideas (the ZERO) to tangible short-term results that provide validating signal and a path to scale (the ONE). We work with our customers to develop and create net-new opportunities by relentlessly scouring all of Amazon and finding new and innovative ways to strengthen and/or accelerate the Amazon Flywheel.CPT seeks an Applied Science intern to work with a diverse, cross-functional team to build new, innovative customer experiences. Within CPT, you will apply both traditional and novel scientific approaches to solve and scale problems and solutions. We are a team where science meets application. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems.
US, IL, Chicago
MULTIPLE POSITIONS AVAILABLECompany: AMAZON.COM SERVICES LLCPosition Title: Data Scientist ILocation: Chicago, IllinoisPosition Responsibilities:Build the core intelligence, insights, and algorithms that support the real estate acquisition strategies for Amazon physical stores. Tackle cutting-edge, complex problems such as predicting the optimal location for new Amazon stores by bringing together numerous data assets, and using best-in-class modeling solutions to extract the most information out of them. Work with business stakeholders, software development engineers, and other data scientists across multiple teams to develop innovative solutions at massive scale.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Note that this posting is for a handful of teams within Amazon Robotics. Teams include: Robotics, Computer Vision, Machine Learning, Optimization, and more.Are you excited about building high-performance robotic systems that can perceive and learn to help deliver for customers? The Amazon Robotics team is creating new science products and technologies that make this possible, at Amazon scale. We work at the intersection of computer vision, machine learning, robotic manipulation, navigation, and human-robot interaction.Amazon Robotics is seeking broad, curious applied scientists and engineering interns to join our diverse, full-stack team. In addition to designing, building, and delivering end-to-end robotic systems, our team is responsible for core infrastructure and tools that serve as the backbone of our robotic applications, enabling roboticists, applied scientists, software and hardware engineers to collaborate and deploy systems in the lab and in the field. We will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Come join us!A day in the lifeAs an intern you will develop a new algorithm to solve one of the challenging computer vision and manipulation problems in Amazon's robotic warehouses. Your project will fit your academic research experience and interests. You will code and test out your solutions in increasingly realistic scenarios and iterate on the idea with your mentor to find the best solution to the problem.
US, WA, Seattle
Are you excited about building high-performance robotic systems that can perceive, learn, and act intelligently alongside humans? The Robotics AI team is creating new science products and technologies that make this possible, at Amazon scale. We work at the intersection of computer vision, machine learning, robotic manipulation, navigation, and human-robot interaction.The Amazon Robotics team is seeking broad, curious applied scientists and engineering interns to join our diverse, full-stack team. In addition to designing, building, and delivering end-to-end robotic systems, our team is responsible for core infrastructure and tools that serve as the backbone of our robotic applications, enabling roboticists, applied scientists, software and hardware engineers to collaborate and deploy systems in the lab and in the field. Come join us!
US, WA, Bellevue
Employer: Amazon.com Services LLCPosition: Research Scientist IILocation: Bellevue, WA Multiple Positions Available1. Research, build and implement highly effective and innovative methods in Statistical Modeling, Machine Learning, and other quantitative techniques such as operational research and optimization to deliver algorithms that solve real business problems.2. Take initiative to scope and plan research projects based on roadmap of business owners and enable data-driven solutions. Participate in shaping roadmap for the research team.3. Ensure data quality throughout all stages of acquisition and processing of the data, including such areas as data sourcing/collection, ground truth generation, data analysis, experiment, evaluation and visualization etc.4. Navigate a variety of data sources, understand the business reality behind large-scale data and develop meaningful science solutions.5. Partner closely with product or/and program owners, as well as scientists and engineers in cross-functional teams with a clear path to business impact and deliver on demanding projects.6. Present proposals and results in a clear manner backed by data and coupled with conclusions to business customers and leadership team with various levels of technical knowledge, educating them about underlying systems, as well as sharing insights.7. Perform experiments to validate the feature additions as requested by domain expert teams.8. Some telecommuting benefits available.The pay range for this position in Bellevue, WA is $136,000-$184,000 (yr); however, base pay offered may vary depending on job-related knowledge, skills, and experience. A sign-on bonus and restricted stock units may be provided as part of the compensation package, in addition to a full range of medical, financial, and/or other benefits, dependent on the position offered. This information is provided by the Washington Equal Pay Act. Base pay information is based on market location. Applicants should apply via Amazon's internal or external careers site.#0000
US, VA, Arlington
The Central Science Team within Amazon’s People Experience and Technology org (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal. As Director for PXT Central Science Technology, you will be responsible for leading multiple teams through rapidly evolving complex demands and define, develop, deliver and execute on our science roadmap and vision. You will provide thought leadership to scientists and engineers to invent and implement scalable machine learning recommendations and data driven algorithms supporting flexible UI frameworks. You will manage and be responsible for delivering some of our most strategic technical initiatives. You will design, develop and operate new, highly scalable software systems that support Amazon’s efforts to be Earth’s Best Employer and have a significant impact on Amazon’s commitment to our employees and communities where we both serve and employ 1.3 million Amazonians. As Director of Applied Science, you will be part of the larger technical leadership community at Amazon. This community forms the backbone of the company, plays a critical role in the broad business planning, works closely with senior executives to develop business targets and resource requirements, influences our long-term technical and business strategy, helps hire and develop engineering leaders and developers, and ultimately enables us to deliver engineering innovations.This role is posted for Arlington, VA, but we are flexible on location at many of our offices in the US and Canada.