Detección de pronunciación para la nueva experiencia de aprendizaje de inglés de Alexa

El aumento de datos, novedosas funciones de pérdidas y un entrenamiento con poca supervisión permiten crear un modelo innovador para detectar errores de pronunciación.

En enero de 2023, Alexa lanzó en España una experiencia de aprendizaje de idiomas para ayudar a los hispanohablantes a aprender inglés para principiantes. Esta experiencia se desarrolló en colaboración con Vaughan, el principal proveedor de aprendizaje de inglés en España, con el objetivo de ofrecer un programa de aprendizaje de inglés inmersivo centrado en la mejora de la pronunciación.

Ahora estamos ampliando esta oferta a México y a la población de habla hispana de Estados Unidos, y en el futuro planeamos añadir más idiomas. Esta experiencia de aprendizaje de idiomas incluye lecciones estructuradas de vocabulario, gramática, expresión y pronunciación, con ejercicios prácticos y pruebas. Para probarla, configura el idioma de tu dispositivo a español y dile a Alexa "Quiero aprender inglés".

Mini-lesson content page_ES.png
Página de contenidos de lecciones cortas: lecciones de vocabulario, gramática, expresión y pronunciación.

Lo más destacado de esta skill de Alexa es su función de pronunciación, la cual proporciona información precisa cada vez que un cliente pronuncia mal una palabra o frase. En la Conferencia Internacional de Acústica, Habla y Procesamiento de Señales (ICASSP por sus siglas en inglés) de este año, presentamos un artículo en el que describíamos nuestro innovador método de detección de errores de pronunciación.

alexaspeechspanish.jpg
Corrección de pronunciación: El texto en azul indica una pronunciación correcta, mientras que el rojo indica una pronunciación incorrecta. Para frases/palabras pronunciadas incorrectamente, Alexa brindará instrucciones detalladas sobre cómo pronunciarlas.

Nuestro método utiliza un novedoso modelo fonético de redes neuronales recurrentes (RNN-T por sus siglas en inglés) que predice los fonemas, las unidades más pequeñas del habla, a partir de la pronunciación del alumno. Por lo tanto, el modelo puede proporcionar una evaluación detallada de la pronunciación a nivel de palabra, sílaba o fonema. Por ejemplo, si un alumno pronuncia incorrectamente la palabra "rabbit" como "rabid", el modelo mostrará la secuencia de cinco fonemas R AE B IH D. Posteriormente, puede detectar los fonemas (IH D) y la sílaba (-bid) mal pronunciados utilizando la alineación de Levenshtein para comparar la secuencia de fonemas con la secuencia de referencia "R AE B AH T".

El artículo destaca dos brechas de conocimiento que no se habían abordado en anteriores modelos de pronunciación. La primera es la capacidad de distinguir fonemas similares en distintos idiomas (por ejemplo, la "r" rodada en español vs. la "r" en inglés). Para ello, diseñamos un léxico de pronunciación multilingüe y creamos un inmenso conjunto de datos fonéticos mixtos para el programa de aprendizaje.

La otra brecha de conocimiento es la capacidad de aprender patrones únicos de pronunciación errónea de los alumnos de idiomas. Para ello, aprovechamos la autorregresividad del modelo RNN-T, es decir, la dependencia de sus resultados de las entradas y salidas anteriores. Este conocimiento del contexto significa que el modelo puede captar patrones frecuentes de pronunciación errónea a partir de los datos del entrenamiento. Nuestro modelo de pronunciación ha obtenido los mejores resultados tanto en precisión de predicción de fonemas, como de detección de errores de pronunciación.

Aumento de datos L2

Uno de los principales retos técnicos a la hora de crear un modelo de reconocimiento fonético para hablantes no nativos (L2) es que los conjuntos de datos para el diagnóstico de errores de pronunciación son muy limitados. En nuestro artículo de Interspeech 2022 "L2-GEN: Un enfoque neuronal de parafraseo de fonemas para el diagnóstico de errores de pronunciación en síntesis del habla L2", planteamos cerrar esta brecha mediante el incremento de datos. En concreto, creamos un parafraseador de fonemas que puede generar fonemas realistas de L2 para hablantes de un lugar específico, por ejemplo, fonemas que representen a un hablante nativo de español hablando en inglés.

Como es habitual en las tareas de corrección de errores gramaticales, utilizamos un modelo de secuencia a secuencia, pero invertimos la dirección de la tarea entrenando al modelo para pronunciar mal las palabras en lugar de corregir los errores de pronunciación. Además, para enriquecer y diversificar aún más las secuencias de fonemas L2 generados, propusimos un componente de decodificación diversificado y consciente de las preferencias que combina una búsqueda en haz diversificada con una pérdida de preferencia que se inclina hacia los errores de pronunciación similares a los humanos.

Para cada tono de entrada o fragmento del habla, el modelo produce varios fonemas posibles como salidas, y las secuencias de fonemas se modelan como un árbol en el que las posibilidades proliferan con cada nuevo tono. Normalmente, las secuencias de fonemas mejor clasificadas se extraen del árbol mediante las búsquedas en haz que persigue solo las ramas del árbol con las probabilidades más altas. En nuestro trabajo, sin embargo, propusimos un método de búsqueda en haz que da prioridad a los fonemas inusuales, o candidatos a fonema que difieren de la mayoría de los demás en la misma profundidad del árbol.

A partir de fuentes establecidas en la documentación sobre aprendizaje de idiomas, también elaboramos listas de errores de pronunciación comunes a nivel de fonema, representados como pares de fonemas, uno del fonema estándar de la lengua y otro de su variante no estándar. Construimos una función de pérdida que, durante el proceso de aprendizaje del modelo, da prioridad a los resultados que utilizan las variantes no estándar de nuestra lista.

En los experimentos observamos mejoras de precisión de hasta el 5% en la detección de errores de pronunciación con respecto a un modelo de referencia entrenado sin datos adicionales.

Equilibrando el falso rechazo y la falsa aceptación

Una consideración clave a la hora de diseñar un modelo de pronunciación para una experiencia de aprendizaje de idiomas es equilibrar la proporción de falsos rechazos y falsas aceptaciones. Un falso rechazo se produce cuando el modelo de pronunciación detecta un error de pronunciación, pero en realidad el alumno estaba en lo cierto o utilizaba una pronunciación coherente pero ligeramente acentuada. Una falsa aceptación se produce cuando un alumno pronuncia mal una palabra y el modelo no lo detecta.

Nuestro sistema tiene dos características de diseño enfocadas a equilibrar estas dos métricas. Para reducir las falsas aceptaciones, primero combinamos nuestros léxicos de pronunciación estándar para inglés y español en un léxico único con múltiples fonemas correspondientes a cada palabra. Después, utilizamos ese léxico para analizar automáticamente muestras de habla no comentadas que se clasifican en tres categorías: español nativo, inglés nativo y español e inglés codificados. El entrenamiento del modelo con este conjunto de datos le permite distinguir diferencias muy sutiles entre fonemas.

Para reducir los falsos rechazos utilizamos un léxico de pronunciación multirreferencial en el que cada palabra se asocia a varias pronunciaciones de referencia. Por ejemplo, la palabra "data" puede pronunciarse como "day-tah" o "dah-tah" y el sistema aceptará ambas variaciones como correctas.

Actualmente seguimos estudiando varios métodos para mejorar nuestra función de evaluación de la pronunciación. Uno de ellos es la creación de un modelo multilingüe que pueda utilizarse para evaluar la pronunciación en muchos idiomas. También estamos ampliando el modelo para diagnosticar más características de pronunciación errónea, como el tono y el acento léxico.

Related content

US, VA, Herndon
Do you love decomposing problems to develop machine learning (ML) products that impact millions of people around the world? Would you enjoy identifying, defining, and building ML software solutions that revolutionize how businesses operate? The Global Practice Organization in Professional Services at Amazon Web Services (AWS) is looking for a Software Development Engineer II to build, deliver, and maintain complex ML products that delight our customers and raise our performance bar. You’ll design fault-tolerant systems that run at massive scale as we continue to innovate best-in-class services and applications in the AWS Cloud. Key job responsibilities Our ML Engineers collaborate across diverse teams, projects, and environments to have a firsthand impact on our global customer base. You’ll bring a passion for the intersection of software development with generative AI and machine learning. You’ll also: - Solve complex technical problems, often ones not solved before, at every layer of the stack. - Design, implement, test, deploy and maintain innovative ML solutions to transform service performance, durability, cost, and security. - Build high-quality, highly available, always-on products. - Research implementations that deliver the best possible experiences for customers. A day in the life As you design and code solutions to help our team drive efficiencies in ML architecture, you’ll create metrics, implement automation and other improvements, and resolve the root cause of software defects. You’ll also: - Build high-impact ML solutions to deliver to our large customer base. - Participate in design discussions, code review, and communicate with internal and external stakeholders. - Work cross-functionally to help drive business solutions with your technical input. - Work in a startup-like development environment, where you’re always working on the most important stuff. About the team The Global Practice Organization for Analytics is a team inside the AWS Professional Services Organization. Our mission in the Global Practice Organization is to be at the forefront of defining machine learning domain strategy, and ensuring the scale of Professional Services' delivery. We define strategic initiatives, provide domain expertise, and oversee the development of high-quality, repeatable offerings that accelerate customer outcomes. Inclusive Team Culture Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Work/Life Balance Our team puts a high value on work-life harmony. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. As such, this is a customer facing role in a hybrid delivery model. Project engagements include remote delivery methods and onsite engagement that will include travel to customer locations as needed. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future. This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed. We are open to hiring candidates to work out of one of the following locations: Atlanta, GA, USA | Austin, TX, USA | Boston, MA, USA | Chicago, IL, USA | Herndon, VA, USA | Minneapolis, MN, USA | New York, NC, USA | San Diego, CA, USA | San Francisco, CA, USA | Seattle, WA, USA
US, MA, North Reading
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics We are open to hiring candidates to work out of one of the following locations: North Reading, MA, USA | Seattle, WA, USA | Westborough, MA, USA
CA, BC, Vancouver
Amazon Web Services (AWS) is building a world-class marketing organization that drives awareness and customer engagement with the goal of educating developers, IT and line-of-business professionals, startups, partners, and executive decision makers about AWS services and solutions, their benefits, and differentiation. As the central data and science organization in AWS Marketing, the Data: Science and Engineering (D:SE) team builds measurement products, AI/ML models for targeting, and self-service insights capabilities for AWS Marketing to drive better measurement and personalization, improve data access and analytical self-service, and empower strategic data-driven decisions. We work globally as a central team and establish standards, benchmarks, and best practices for use throughout AWS Marketing. We are looking for a Principal Data Scientist with deep expertise in scaling measurement science, content ranking and rapid experimentation at scale, with strong interest in building scalable solutions in partnership with our engineering organization. You will lead strategic measurement science initiatives across AWS Marketing & Sales ranging anywhere between recommender engines, scaling experimentation and measurement science, real-time inference, and cross-channel orchestration. You are an hands-on innovator who can contribute to advancing Marketing measurement technology in a B2B environment, and push the limits on what’s scientifically possible with a razor sharp focus on measurable customer and business impact. You will work with recognized B2B Marketing Science and AI/ML experts to develop large-scale, high-performing measurement science models and AI/ML capabilities. We are at a pivotal moment in our organization where AI/ML and measurement velocity has reached an unseen momentum, and we need to scale fast in order to maintain it. Your work will be a key input into a few of our key business goals. You will advance the state of the art in measurement at scale. We are open to hiring candidates to work out of one of the following locations: Vancouver, BC, CAN
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best. Key job responsibilities • Develop automated laboratory workflows. • Perform data QC, document results, and communicate to stakeholders. • Maintain updated understanding and knowledge of methods. • Identify and escalate equipment malfunctions; troubleshoot common errors. • Participate in the updating of protocols and database to accurately reflect the current practices. • Maintain equipment and instruments in good operating condition • Adapt to unexpected schedule changes and respond to emergency situations, as needed. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, VA, Arlington
Amazon’s mission is to be the most customer centric company in the world. The Workforce Staffing (WFS) organization is on the front line of that mission by hiring the hourly fulfillment associates who make that mission a reality. To drive the necessary growth and continued scale of Amazon’s associate needs within a constrained employment environment, Amazon has created the Workforce Intelligence (WFI) team. This team will (re)invent how Amazon attracts, communicates with, and ultimately hires its hourly associates. This team owns multi-layered research and program implementation to drive deep learning, process improvements, and strategic recommendations to global leadership. Are you passionate about data? Do you enjoy questioning the status quo? Do complex and difficult challenges excite you? If yes, this may be the team for you. The Data Scientist will be responsible for creating cutting edge algorithms, predictive and prescriptive models as well as required data models to facilitate WFS at-scale warehouse associate hiring. This role acts as an internal consultant to the marketing, biz ops and candidate experience teams covering responsibilities such as at-scale hiring process improvement, analyzing large scale candidate/associate data and being strategic to providing best candidate hiring experience to WFS warehouse associate candidates. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Seattle
Are you excited about developing generative AI and foundation models to revolutionize automation, robotics and computer vision? Are you looking for opportunities to build and deploy them on real problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience - at Amazon scale. We are looking for scientists, engineers and program managers for a variety of roles. The Amazon Robotics software team is seeking a Applied Scientist to focus on large vision and manipulation machine learning models. This includes building multi-viewpoint and time-series computer vision systems. It includes using machine learning to drive hardware movement. It includes building large-scale models using data from many different tasks and scenes. This work spans from basic research such as cross domain training, to experimenting on prototype in the lab, to running wide-scale A/B tests on robots in our facilities. Key job responsibilities * Research vision - Where should we be focusing our efforts * Research delivery – Proving/dis-proving strategies in offline data or in the lab * Production studies - Insights from production data or ad-hoc experimentation. About the team This team invents and runs robots focused on grasping and packing items. These are typically 6-dof style robotic arms. Our work ranges from the long-term-research on basic science to deploying/supporting large production fleets handling billions of items per year. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, VA, Arlington
Amazon launched the Generative AI (GenAI) Innovation Center (GAIIC) in Jun 2023 to help AWS customers accelerate enterprise innovation and success with Generative AI (https://press.aboutamazon.com/2023/6/aws-announces-generative-ai-innovation-center). Customers such as Highspot, Lonely Planet, Ryanair, and Twilio are engaging with the GAI Innovation Center to explore developing generative solutions. GAIIC provides opportunities to innovate in a fast-paced organization that contributes to game-changing projects and technologies that get deployed on devices and in the cloud. As a data scientist at GAIIC, you are proficient in designing and developing advanced Generative AI based solutions to solve diverse customer problems. You will be working with terabytes of text, images, and other types of data to solve real-world problems through Gen AI. You will be working closely with account teams and ML strategists to define the use case, and with other scientists and ML engineers on the team to design experiments, and find new ways to deliver value to the customer. The successful candidate will possess both technical and customer-facing skills that will allow you to be the technical “face” of AWS within our solution providers’ ecosystem/environment as well as directly to end customers. You will be able to drive discussions with senior technical and management personnel within customers and partners. This position requires that the candidate selected be a US Citizen and currently possess and maintain an active Top Secret security clearance. About the team Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Denver, CO, USA
US, CA, Sunnyvale
Are you passionate about solving unique customer-facing problem at Amazon scale? Are you excited by developing and productionizing machine learning, deep learning algorithms and leveraging tons of Amazon data to learn and infer customer shopping patterns? Do you enjoy working with a diverse set of engineers, machine learning scientists, product managers and user-experience designers? If so, you have found the right match! Virtual Try On (VTO) at Amazon Fashion & Fitness is looking for an exceptional Applied Scientist to join us to build our next generation virtual try on experience. Our goal is to help customers evaluate how products will fit and flatter their unique self before they ship, transforming customers' shopping into a personalized journey of inspiration, discovery, and evaluation. In this role, you will be responsible for building scalable computer vision and machine learning (CVML) models, and automating their application and expansion to power customer-facing features. Key job responsibilities - Tackle ambiguous problems in Computer Vision and Machine Learning, and drive full life-cycle of CV/ML projects. - Build Computer Vision, Machine Learning and Generative AI models, perform proof-of-concept, experiment, optimize, and deploy your models into production. - Investigate and solve exciting and difficult challenges in Image Generation, 3D Computer Vision, Generative AI, Image Understanding and Deep Learning. - Run A/B experiments, gather data, and perform statistical tests. - Lead development and productionalization of CV, ML, and Gen AI models and algorithms by working across teams. Deliver end to end. - Act as a mentor to other scientists on the team. We are open to hiring candidates to work out of one of the following locations: Sunnyvale, CA, USA
US, CA, Sunnyvale
At Amazon Fashion, we are obsessed with making Amazon Fashion the most loved fashion destinations globally. We're searching for Computer Vision pioneers who are passionate about technology, innovation, and customer experience, and who are enthusiastic about making a lasting impact on the industry. You'll be working with talented scientists, engineers, and product managers to innovate on behalf of our customers. If you're fired up about being part of a dynamic, driven team, then this is your moment to join us on this exciting journey and change the world of eCommerce forever Key job responsibilities As a Applied Scientist, you will be at the forefront to define, own and drive the science that span multiple machine learning models and enabling multiple product/engineering teams and organizations. You will partner with product management and technical leadership to identify opportunities to innovate customer facing experiences. You will identify new areas of investment and work to align product roadmaps to deliver on these opportunities. As a science leader, you will not only develop unique scientific solutions, but more importantly influence strategy and outcomes across different Amazon organizations such as Search, Personalization and more. This role is inherently cross-functional and requires a strong ability to communicate, influence and earn the trust of software engineers, technical and business leadership. We are open to hiring candidates to work out of one of the following locations: Sunnyvale, CA, USA