How to teach Transformers to care about word order

New position encoding scheme improves state-of-the-art performance on several natural-language-processing tasks.

The Transformer is a neural-network architecture that has proven extremely useful for natural-language-processing tasks because it can recognize long-range dependencies. It could, for instance, recognize that in a sentence that includes the word “rented”, the word “flat” is more likely to mean “apartment” than it would be otherwise, even if “rented” is the second word in the sentence and “flat” the 10th.

In its most basic form, the Transformer is indifferent to word order. It can recognize the relationship between “rented” and “flat”, but it doesn’t care which comes first.

Word order, however, can make a big difference to meaning. Consider, for instance, the sentences “We rented a small but clean, well-equipped two-bed flat” and “We rented a small but clean, well-equipped flat-bed truck”.

Position embedding.png
These images map 252 words of an input text sequence (y-axis) against the 512 latent position features identified by two different position-encoding schemes. Lighter colors indicate higher values for features, darker colors lower values. FLOATER (bottom) produces a more regular encoding than an earlier scheme (top), which also learns its position feature set from training data. The vertical lines toward the bottom of the top visualization indicate that the encoding model is simply using the same encoding for input sequences longer than those it saw during training, while FLOATER’s smooth gradation from light to dark demonstrates that its encoding generalizes easily to longer sequences.

Starting with the paper that introduced the Transformer, researchers have proposed a series of position encoders that inject word-order information into the Transformer model. But last week, at the International Conference on Machine Learning, we presented a new position encoder that enables better performance than its predecessors on a range of natural-language-processing (NLP) tasks.

We designed our position encoder so that it can be integrated into existing Transformer models, conferring its benefits to NLP systems that already have been trained extensively on large data sets.

Before the Transformer was introduced in 2017, the most popular architecture for NLP was the long short-term memory, or LSTM. LSTMs process sequenced inputs in order, and each output reflects both the inputs and the outputs that preceded it.

LSTMs are very good at inferring local relationships — a word’s relationships, both syntactic and semantic, with the two or three words that immediately precede it — but they’re not as good at modeling long-range dependencies. That’s where the Transformer excels.

Position encodings are an attempt to achieve the best of both worlds: an awareness of long-range dependencies and a sensitivity to local word order. The ideal position encoding should have three properties:

  1. It should be able to handle sequences of arbitrary length; that is, it shouldn’t be locked in to some maximum sequence length.
  2. It should be learnable from training data; different encodings may work better for different tasks.
  3. It should be efficient; adding position encoding shouldn’t unreasonably inflate the size of the neural model.

Past position encoding schemes have met at best two of these criteria. For instance, the original Transformer paper proposed an encoding based on a family of sinusoidal functions; that encoding remains popular, but it is not learnable.

Our scheme, which we call FLOATER, is the first to meet all three criteria.

The naïve way to encode position would be simply to assign successive numbers to successive words in an input sequence. But this has drawbacks in a machine learning context. If at runtime the model sees a sequence of a length it did not encounter during training, it will be flummoxed about how to proceed.

So most position encoding schemes instead use position vectors, which carry information that can be used to deduce the relative positions of two inputs. If those schemes are fully learnable, however, they tend to inflate the model size; or, to keep model inflation under control, they limit the distances across which relative position can be compared.

Functional approach

Instead of learning to directly compute a position vector from each word in an input sequence, FLOATER learns a function that computes each word’s position vector from that of the word that preceded it.

Learning a general function rather than direct mappings makes FLOATER much more space efficient than other learnable encoding schemes. But a general function can also be applied to any word in a sequence, regardless of its position, so FLOATER is indifferent to sequence length.

Any given manually engineered position function — such as the sinusoidal functions proposed in the original Transformer paper — can be thought of as a special case of the general FLOATER function. So in a pretrained network, we can simply substitute FLOATER for any such function and then fine-tune it on a small set of training data.

Past work on position encoding has shown that re-encoding position information at every layer of a Transformer network improves performance on NLP tasks. If we allowed FLOATER to learn a different function for every layer, the model size would again begin to inflate.

So instead, we learn a single function that is applied at every layer. This results in different position encodings at each layer, however, because the inputs are different. Our experiments indicate that this approach strikes a good balance between model size and performance improvements.

In one set of experiments, we compared our position encoder to its two leading predecessors on four different machine translation tasks and found that it delivered the best results across the board.

In another set of experiments, we added our position encoder to Transformer models that had previously been trained on three different language-understanding and question-answering tasks.

Of 23 distinct tasks, the addition of our position encoder improved performance on 21. The two on which its performance fell slightly short were low-data versions of tasks on which, with larger sets of training data, it improved performance.

About the Author
Hsiang-Fu Yu is a senior applied scientist in Amazon's Search Science and AI organization.

Related content

Work with us

See more jobs
US, WA, Seattle
Come join Amazon Halo's Computer Vision and Machine Learning Science team!We are looking for an Applied Scientist passionate about improving customers’ health and wellness to join our growing CVML science team.As an Applied Scientist, you will be a member of an algorithm team responsible for the end-end delivery of CVML-based algorithms.Job responsibilities· Research, develop, implement and evaluate novel CVML algorithms· Work on large-scale datasets, focusing on creating scalable and accurate CVML systems in versatile application fields· Design data collection experiments, perform statistical analysis and deliver ML models.· Collaborate closely with team members on developing systems from prototyping to production level· Work closely with software engineering teams to drive scalable, real-time implementations· Track general business activity and provide clear, compelling management reports on a regular basisAmazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, WA, Bellevue
We are building a first of its kind system to optimize several diverse voice and visual experiences spanning Amazon Echo products. Our solutions are solving key challenges in how customers learn about and interact with voice-first devices, and enable customers to get the most out of our products.A day in the lifeAs a Sr. Manager, Applied Science, you will work on problems spanning dialog systems, personalization, contextual bandits, recommender systems, and reinforcement learning, developing novel solutions using large-scale real-world datasets. You will develop innovative solutions that enable Alexa to learn from past customer interactions to drive future engagement and utility. You are responsible for setting the research roadmap for a team of applied and research scientists, and develop models and approaches that touch key customers experiences for Amazon Alexa customers. You have the ability to influence the long-term direction of the Alexa voice experience.About the hiring groupThis is a rare and exciting opportunity to be a trailblazer at the intersection of cutting edge science and engineering to build ML-based products that impact the lives of millions of customers. You will be responsible for defining critical research questions, developing innovative solutions to large-scale business problems, publishing results, and working with engineering teams to evaluate and deploy these solutions. You will be a strategic leader contributing to software architecture design, long term product strategies, and business goal definition and prioritization.Job responsibilitiesHelp us shape the future of Alexa where it not only responds to requests you make but also proactively anticipates what you might need next and informs you of it at the right moment. We are looking for a seasoned science leader to lead of team of applied scientists, research scientists, and language engineers to optimize several diverse voice and visual experiences across Amazon Echo products. As part of the Alexa Experience science team, you will be solving key challenges in how customers learn about and interact with voice assistants, and enable customers to get the most out of our products.You will lead a team of scientists who will work on problems spanning dialog systems, personalization, contextual bandits, recommender systems, and reinforcement learning, developing novel solutions using large-scale real-world datasets. You will partner closely with engineering and product teams to build, evaluate, and deliver your solutions. You will develop innovative solutions that enable Alexa to learn from past customer interactions to drive future engagement and utility. You are responsible for leading and setting the research and development roadmap for a team of applied and research scientists, and develop models and approaches that touch key customers experiences for Amazon Alexa customers. You have the ability to influence the long-term direction of the Alexa voice experience, and will be a strategic leader contributing to software architecture design, long term product strategies, and business goal definition and prioritization.As Senior Manager, Applied Science you will:Lead a team of scientists and analysts in the design, development, test, deployment and operation of machine learning models.Contribute directly to our growth by hiring smart and motivated scientists and software developers that can apply state of the art machine learning algorithms to deliver innovative features for our customers.Be responsible for ensuring sanity of algorithms, architecture, operational excellence and quality, delivering results on time.Work closely with other teams across Amazon to deliver platform features that require cross-team leadership.Mentor and lead as a manager with a direct handle on the pulse of each employee on your team.Regularly dive into learning algorithms, evaluation metrics, model architecture, code, test plans, project plans, deployments and operations to drive excellence through your organization.Represent your business and operations to senior leaders within Amazon.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, MA, Cambridge
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.com Services, LLCTitle: Applied Scientist IILocation: Cambridge, MAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Come join Amazon Halo's Computer Vision and Machine Learning Science team!We are looking for an Applied Scientist passionate about improving customers’ health and wellness to join our growing CVML science team.As an Applied Scientist, you will be a member of an algorithm team responsible for the end-end delivery of CVML-based algorithms.Job responsibilities· Research, develop, implement and evaluate novel CVML algorithms· Work on large-scale datasets, focusing on creating scalable and accurate CVML systems in versatile application fields· Design data collection experiments, perform statistical analysis and deliver ML models.· Collaborate closely with team members on developing systems from prototyping to production level· Work closely with software engineering teams to drive scalable, real-time implementations· Track general business activity and provide clear, compelling management reports on a regular basisAmazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, WA, Seattle
Amazon Business (www.amazon.com/business) is an online store that combines the selection, convenience and value customers have come to know and love from Amazon, with new features and unique benefits tailored to the needs of businesses. Amazon Business provides easy access to hundreds of millions of products – everything from IT and lab equipment to education and food-service supplies. Amazon Business customers also enjoy a variety of benefits, including business-only pricing and selection, a multi-seller marketplace, single- or multi-user business accounts, approval workflow, purchasing system integrations, payment solutions, tax exemptions, dedicated customer support, Business Prime and more.Business Prime is a paid subscription service for verified business, charity or government organizations that provides the familiar Prime shopping experience end users love at home, with pricing plans and benefits that are suited for work. Business Prime now serves over a million employees in organizations throughout the world. Our customers range from government entities with tens of thousands of users to sole proprietors.The Business Prime team is looking for an experienced and motivated Data Scientist to generate data-driven insights influencing the Business Prime direction, build the necessary predictive models, optimization algorithms and customer behavioral segments allowing us to discover and expand the value proposition of Business Prime to the customers. Specific responsibilities include:- Customer lifecycle analysis improving targeting, customer identification, and spend behavior.- Data driven insights to accelerate acquisition of new members.- Grow benefits adoption based on customer segment, vertical, and drive customers to their "aha moment".- Predict customers at risk of churn and declining engagement.- Identify ineligible accounts including fraud, abuse, and other undesired behaviors.- Experiments to enhance membership value for B2B customers.In this role, you will be a technical expert with significant scope and impact. You will work with Business Intelligence Engineers, Financial Analysts, Product Managers, Software Engineers, Data Engineers, and other Data Scientists, to build new and enhance existing ML models to optimize customer experience. The successful Data Scientist will have extreme bias for action needed in a startup environment, with outstanding leadership skills, proven ability to build and manage medium-scale modeling projects, identify data requirements, build methodology and tools that are statistically grounded. We are seeking someone who can thrive in a fast-paced, high-energy and fun work environment where we deliver value incrementally and frequently. We value highly technical people who know their subject matter deeply and are willing to learn new areas. We look for individuals who know how to deliver results and show a desire to develop themselves, their colleagues, and their career.
US, WA, Seattle
Amazon strives to be Earth's most customer-centric company. To achieve this, we hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment.Economists for Amazon will be expected to work directly with senior management on key business problems faced in retail and marketing. The role allows you to influence company-wide strategy by applying the frontier of economic thinking to forecasting, program evaluation, customer behavior and other areas. Economists at Amazon will be expected to develop new techniques to process large data sets, use econometric models and methods to address complex problems, design and test hypothesis on customer pain points and key growth ideas and contribute to design of automated systems around the company. Economists in our team will work closely with other research scientists, machine learning experts, and economists globally to design new frameworks for understanding business outcome and what impacts customer experiences. Our economists and scientists work closely with software engineers to put algorithms into practice.We are an interdisciplinary team on the cutting edge of economics, statistical analysis, and machine learning whose mission is to solve AI and ML problems that have high risk with high returns Our team seeks an outstanding econometrician with demonstrated experience tackling large-scale time-series forecasting challenges. We prize creative problem solvers with the ability to adapt and extend cutting-edge forecasting models to the unique and interesting problems we have in measuring customer experiences and Amazon business outcome. Our problems include forecasting key customer experiences metrics at different levels of granularities and explaining the drivers of the changes in these metrics. The ideal candidate combines acumen in statistics and applied time-series modeling to grapple with these and other challenges and guide decision-making at the highest levels of accuracy and model explain-ability.
US, CA, Cupertino
Amazon's Simple Storage Service (S3) offers industry-leading scalability, data availability, security, and performance. S3’s Automated Reasoning Group (S3-ARG) develops and applies automated reasoning techniques to deliver correct, secure, durable, and available distributed systems and storage services. S3 is complex: it consists of over 300 microservices each of which is a distributed system. With a very large number of servers and 10s of millions of requests per second, it is also highly concurrent. S3 has a sizable codebase, developed and maintained by a large team of engineers. Getting such a complex and fast-evolving system right requires developing cutting edge automated reasoning methods that are continuously integrated into the software development process. This is what our team does.We work on techniques ranging from deductive proofs to model checking, from static analysis to runtime verification of protocols. We work both at the design and the code level, and connecting the two is essential for us. We partner with development teams to make sure that our methods are deployed across S3 and that correctness is maintained as the software evolves. We have had significant success with adoption and we are key contributors to recent S3 launches such as strong consistency. We are developing innovative methods all the time. We publish our results at conferences and journals.We are a diverse team and are looking for teammates who are enthusiastic to work on these problems and further the state of the art with us. We are seeking candidates who are deep in one area of expertise but also broad enough to take on the most complex cloud computing challenges. If you are interested in exploring, please e-mail s3-arg-jobs@amazon.com.About UsInclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, WA, Seattle
At Alexa Shopping, we strive to enable shopping in everyday life. We allow customers to instantly order whatever they need, by simply interacting with their Smart Devices such as Amazon Show, Spot, Echo, Dot or Tap. Our Services allow you to shop, no matter where you are or what you are doing, you can go from 'I want that' to 'that's on the way' in a matter of seconds. We are seeking the industry's best to help us create new ways to interact, search and shop. Join us, and you'll be taking part in changing the future of everyday lifeWe are seeking a Data Scientist to be part of the NLU science team for Alexa Shopping. This is a strategic role to shape and deliver our technical strategy in developing and deploying NLU, Machine Learning solutions to our hardest customer facing problems. Our goal is to delight customers by providing a conversational interaction. These initiatives are at the heart of the organization and recognized as the innovations that will allow us to build a differentiated product that exceeds customer expectations. We're a high energy, fast growth business excited to have the opportunity to shape Alexa Shopping NLU is defined for years to come. If this role seems like a good fit, please reach out, we'd love to talk to you.This role requires working closely with business, engineering and other scientists within Alexa Shopping and across Amazon to deliver ground breaking features. You will lead high visibility and high impact programs collaborating with various teams across Amazon. You will work with a team of Language Engineers and Scientists to launch new customer facing features and improve the current features.
US, NY, New York
MULTIPLE POSITIONS AVAILABLEEntity: Amazon Web Services, Inc.Title: Data Scientist ILocation: New York, NYPosition Responsibilities:Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLEEntity: Amazon.Com Services LLCTitle: Applied ScientistWorksite: Seattle, WAPosition Responsibilities:Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, CO, Denver
Excited by using massive amounts of data to develop Machine Learning (ML) and Deep Learning (DL) models? Want to help public sector, medical center and non-profit agencies derive business value through the adoption of Artificial Intelligence (AI)? Eager to learn from many different enterprise’s use cases of AWS ML and DL? Thrilled to be key part of Amazon, who has been investing in Machine Learning for decades, pioneering and shaping the world’s AI technology?At Amazon Web Services (AWS), we are helping large enterprises build ML and DL models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. Our Professional Services organization works together with our AWS customers to address their business needs using AI.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML or DL models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers.If you do not live in a market where we have an open Data Scientist position, please feel free to apply. Our Data Scientists can live in any location where we have a Professional Service office.A successful candidate will be a person who enjoys diving deep into data, doing analysis, discovering root causes, and designing long-term solutions. It will be a person who likes to have fun, loves to learn, and wants to innovate in the world of AI. Major responsibilities include:· Understand the customer’s business need and guide them to a solution using our AWS AI Services, AWS AI Platforms, AWS AI Frameworks, and AWS AI EC2 Instances .· Assist customers by being able to deliver a ML / DL project from beginning to end, including understanding the business need, aggregating data, exploring data, building & validating predictive models, and deploying completed models to deliver business impact to the organization.· Use Deep Learning frameworks like PyTorch, Tensorflow, and MxNet to help our customers build DL models.· Use SparkML and Amazon Machine Learning (AML) to help our customers build ML models.· Work with our Professional Services Big Data consultants to analyze, extract, normalize, and label relevant data.· Work with our Professional Services DevOps consultants to help our customers operationalize models after they are built.· Assist customers with identifying model drift and retraining models.· Research and implement novel ML and DL approaches, including using FPGA.· This position can have periods of up to 10% travel.
US, WA, Seattle
At Alexa Shopping, we strive to enable shopping in everyday life. We allow customers to instantly order whatever they need, by simply interacting with their Smart Devices such as Amazon Show, Spot, Echo, Dot or Tap. Our Services allow you to shop, no matter where you are or what you are doing, you can go from 'I want that' to 'that's on the way' in a matter of seconds. We are seeking the industry's best to help us create new ways to interact, search and shop. Join us, and you'll be taking part in changing the future of everyday lifeWe are seeking a Data Scientist to be part of the Correction and Automated Recovery (CARe) team for Alexa Shopping. Our team focuses on solving the hard problem of recovering from shopping errors in the Alexa pipeline and improve customer CX. The solutions will help address unique shopping challenges and maximize the self-learning benefits for shopping. This is a strategic role to shape and deliver our science strategy in developing and deploying Machine Learning solutions to our hardest customer facing problems. The initiatives are at the heart of the organization and recognized as the innovations that are ground breaking and will allow us to exceed customer expectations. If this role seems like a good fit, please reach out, we'd love to talk to you.This role requires working closely with business, engineering and other scientists within Alexa, Shopping and across Amazon to deliver ground breaking features. You will lead high visibility and high impact programs collaborating with various teams across Amazon. You will work with a team of SDEs and Scientists to launch new customer facing features and improve the current features.
US, MA, Cambridge
The Alexa Artificial Intelligence (AI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background, to help build industry-leading Speech and Language technology.About the hiring groupThe Alexa AI team has a mission to push the envelope in Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), and Audio Signal Processing, in order to provide the best-possible experience for our customers.Job responsibilitiesAs an Applied Scientist with the Alexa AI team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, MA, Cambridge
The Alexa Artificial Intelligence (AI) team is looking for a passionate, talented, and inventive Senior Applied Scientist with a strong machine learning background, to help build industry-leading Speech and Language technology.About the hiring groupThe Alexa AI team has a mission to push the envelope in Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), and Audio Signal Processing, in order to provide the best-possible experience for our customers.Job responsibilitiesAs a Senior Applied Scientist with the Alexa AI team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, MA, Cambridge
The Alexa Artificial Intelligence (AI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background, to help build industry-leading Speech and Language technology.We work on privacy initiatives related to Alexa such as continual learning, Federated learning and synthetic data generation.About the hiring groupThe Alexa AI team has a mission to push the envelope in Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), and Audio Signal Processing, in order to provide the best-possible experience for our customers.Job responsibilitiesAs an Applied Scientist with the Alexa AI team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, CA, Santa Monica
Amazon is investing heavily in building a customer centric, world class advertising business across its many unique audio, video, and display surfaces. We are looking for an Applied Scientist who has a deep passion for building machine-learning solutions in our advertising decision system. In this role, you will be on the cutting edge of developing monetization solutions for Live TV, Connected TV and streaming Audio. These are nascent, high growth areas, where advertising monetization is an important, fully integrated part of the core strategy for each business.As an Sr. Data Scientist, you will be tasked with leading innovations in machine learning algorithms to deliver ads across platforms influencing product features and architectural choices for decision making systems. You will need to work with data scientists to invent elegant metrics and associated measurement models, and develop algorithms that help advertisers test and learn the impact of advertising strategies across channels on these metrics while ensuring a great customer experience.Key responsibilities:· Rapidly design, prototype and test machine learning algorithms for optimizing advertising reach, frequency and return on advertising spend· Build systems that extract and process volumes of disparate data using a variety of econometric and machine learning approaches. These systems should be designed to scale with exponential growth in data and run continuously.· Leverage knowledge of advanced software system and algorithm development to build our measurement and optimization engine.· Contribute intellectual property through patent generation.· Functionally decompose complex problems into simple, straight-forward solutions.· Understand system inter-dependencies and limitations as well as analytic inter-dependencies to build efficient solutions.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, WA, Seattle
We’re working on the future. If you are seeking an iterative fast-paced environment where you can drive innovation, apply state-of-the-art technologies to solve real world delivery challenges, and provide visible benefit to customers, this is your opportunity.Come work on the Amazon Prime Air Team!We're looking for an outstanding engineer who combines strong knowledge of aerodynamics with expertise in using CFD simulations to drive vehicle design. As a member of the high-fidelity aero team, you will have a direct hand in shaping our future drone designs and will interface closely with several other teams, including the conceptual design, wind tunnel, and controls teams.Responsibilities will include using CFD to aid in:· Vehicle-level conceptual and detailed design· Propeller design· Aerodynamic database generation· Wind tunnel support and validationDeveloping pre- and post-processing tooling is another key component to the role, so strong programming skills are encouraged.Export License ControlThis position may require a deemed export control license for compliance with applicable laws and regulations. Placement is contingent on Amazon’s ability to apply for and obtain an export control license on your behalf.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
US, VA, Arlington
Excited by using massive amounts of data to develop Machine Learning (ML) and Deep Learning (DL) models? Want to help public sector, medical center, and non-profit customers derive business value through the adoption of Artificial Intelligence (AI)? Eager to learn from many different enterprise’s use cases of AWS ML and DL? Thrilled to be key part of Amazon, who has been investing in Machine Learning for decades, pioneering and shaping the world’s AI technology?At Amazon Web Services (AWS), we are helping large enterprises build ML and DL models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. Our Professional Services organization works together with our AWS customers to address their business needs using AI.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML or DL models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers.If you do not live in a market where we have an open Data Scientist position, please feel free to apply. Our Data Scientists can live in any location where we have a WWPS Professional Service office.We’re looking for top architects, system and software engineers capable of using ML and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.The primary responsibilities of this role are to:· Design data architectures and data lakes· Provide expertise in the development of ETL solutions on AWS· Use ML tools, such as Amazon SageMaker Ground Truth (GT) to annotate data. Work with Professional Services on designing workflow and user interface for GT annotation.· Collaborate with our data scientists to create scalable ML solutions for business problems· Interact with customer directly to understand the business problem, help and aid them in implementation of their ML ecosystem· Analyze and extract relevant information from large amounts of historical data — provide hands-on data wrangling expertise· Work closely with account team, research scientist teams and product engineering teams to drive model implementations and new algorithms· This position can have periods of up to 10% travel.
US
Are you passionate about building successful Data transformations within the Public Sector? At Amazon Web Services (AWS), we’re hiring highly technical Data engineers to collaborate with our customers and partners on key engagements. Our consultants will develop and deliver proof-of-concept projects, technical workshops, and support implementation projects. These professional services engagements will focus on customer solutions such as Data and Analytics, HPC and more.In this role, you will work with our partners, customers and focus on our AWS offerings such Amazon Kinesis, AWS Glue, Amazon Redshift, Amazon EMR, Amazon Athena and more. You will help our customers and partners to remove the constraints that prevent them from leveraging their data to develop business insights.AWS Professional Services engage in a wide variety of projects for customers and partners, providing collective experience from across the AWS customer base and are obsessed about customer success. Our team collaborates across the entire AWS organization to bring access to product and service teams, to get the right solution delivered and drive feature innovation based upon customer needs.You will also have the opportunity to create white papers, writing blogs, build demos and other reusable collateral that can be used by our customers. Most importantly, you will work closely with our Solution Architects, Data Architects and Service Engineering teams.The ideal candidate will have extensive experience with design, development and operations that leverages deep knowledge in the use of services like Amazon Kinesis, Apache Kafka, Apache Spark, Amazon EMR, NoSQL technologies and other 3rd parties.This is a customer facing role. You will be required to travel to client locations and deliver professional services when needed (expected travel time is 20%)Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.
US, WA, Seattle
Would you like to shape the future of the video entertainment industry for movies, TV and live sports events? Does solving complex problems within large scale, production systems excite you? If you answered yes, we have an opportunity for you!Prime Video is disrupting the traditional television and movie industry with a growing library of high-quality media. Prime Video launched in 2007 and has quickly become a strategic priority for the company, reflected in the service’s recent expansion into over 240 countries and territories worldwide.This is a big opportunity to apply Computer Vision and directly impact millions of customers.A day in the lifeIn your day-to-day activities in this role, you'll embrace the challenges of a fast paced market and evolving technologies, and develop Computer Vision and Machine Learning models to extract deep 2/3-D video-understanding of Prime Video content. You will be encouraged to see the big picture, be innovative, and iteratively develop technology to impact millions of our customers. This is a young and evolving business where creativity and drive will have a lasting impact on the way video is enjoyed worldwide.About the hiring groupThe PV-CVML team is a group of Applied Scientists working on a diverse set of 2/3-D video understanding problems while partnering with various teams across Prime Video (PV). The most unique aspect of our team is the broad set of exciting problems we get to work on for our multiple stakeholders across the entire video-streaming vertical. If you want to work on technically cutting-edge problems with massive customer impact, then our team is the perfect fit for you!Job responsibilitiesAs a member of our team, you will apply Computer Vision and Machine Learning to problems that have cross-organizational technological impact. Your work will focus on cleansing and preparing large scale datasets, training and evaluating models and deploying them to production. You will work on large engineering efforts that solve significantly complex problems facing global customers. You will be trusted to operate with independence and are often assigned to focus on areas with significant impact on audience satisfaction. You must be equally comfortable with digging in to customer requirements as you are drilling into design with development teams.We would like you to build models that can perform 2D/3D scene-understanding of all video-content available on Prime Video using computer vision, natural language processing, deep learning and advanced machine learning algorithms. We need to solve problems across many cultures and languages and have a huge amount of human-labelled data as well as operations team to generate labels across many languages to help us achieve these goals. Our team consistently strives to innovate, and holds several novel patents and inventions in the motion picture and television industry. We are highly motivated to extend the state of the art.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.