Exploring cross-lingual transfer learning with unsupervised machine translation

By Chao Wang , Judith Gaspers, Quynh Ngoc Thi Do, Hui Jiang
2021
In Natural Language Understanding (NLU), to facilitate Cross-Lingual Transfer Learning (CLTL), especially CLTL between distant languages, we integrate CLTL with Machine Translation (MT), and thereby propose a novel CLTL model named Translation Aided Language Learner (TALL). TALL is constructed as a standard transformer, where the encoder is a pre-trained multilingual language model. The training of TALL includes an MT-oriented pre-training and an NLU-oriented fine-tuning. To make use of unannotated data, we implement the recently proposed Unsupervised Machine Translation (UMT) technique in the MT-oriented pre-training of TALL. The experimental results show that the application of UMT enables TALL to consistently achieve better CLTL performance than our baseline model, which is the pre-trained multilingual language model serving as the encoder of TALL, without using more annotated data, and the performance gain is relatively prominent in the case of distant languages.

Latest news

US, Virtual
Job summaryAmazon’s High Value Messaging (HVM) Analytics team (part of Customer Behavior Analytics org) is looking for an Applied Scientist to spearhead the rapid growth of our Marketing Measurement solutions. The team focuses on building scalable scientific models to estimate the effectiveness of Amazon marketing efforts and provide actionable insights to the various marketing teams within Amazon. We work closely with business stakeholders and strive to continuously produce tangible impact on their strategic and tactical planning and operations.We are looking for a thought leader that has an aptitude for delivering customer-focused solutions and who enjoys working on the intersection of Big-Data analytics, Machine/Deep Learning, and Causal Inference. A successful candidate will be a self-starter, comfortable with ambiguity, able to think big and be creative, while still paying careful attention to detail. You should be able to translate how data represents the customer journey, be comfortable dealing with large and complex data sets, and have experience using machine learning and/or econometric modeling to solve business problems. You should have strong analytical and communication skills, be able to work with product managers to define key business questions and work with the engineering team to bring our solutions into production. You will join a highly collaborative and diverse working environment that will empower you to shape the future of Amazon marketing, and also allow you to become part of the large science community within the Customer Behavior Analytics (CBA) organization.Key job responsibilities· Apply expertise in ML and causal modeling to develop new models that describe how Amazon’s marketing campaigns impact customers’ actions· Own the end-to-end development of novel scientific models that address the most pressing needs of our business stakeholders and help guide their future actions· Improve upon and simplify our existing solutions and frameworks· Review and audit modeling processes and results for other scientists, both junior and senior· Work with marketing leadership to align our scientific developments with the business strategy· Identify new opportunities that are suggested by the data insights· Bring a department-wide perspective into decision making· Develop and document scientific research to be shared with the greater science community at AmazonAbout the teamThe Customer Behavior Analytics (CBA) organization owns Amazon’s insights pipeline, from data collection to deep analytics. We aspire to be the place where Amazon teams come for answers, a trusted source for data and insights that empower our systems and business leaders to make better decisions. Our outputs shape Amazon product and marketing teams’ decisions and thus how Amazon customers see, use, and value their experience.
US, VA, Arlington
Job summaryAmazon Region Services team is looking for a Senior Data Scientist to lead the research and thought leadership to drive our data and insights strategy for Amazon Region Services. We develop sophisticated algorithms that involve learning from large amounts of data. You will be responsible for researching, experimenting and analyzing predictive and optimization models.You will provide to engineers, direction on technology strategy in data management, insights and Machine learning that will form the nucleus of providing data and insights as a service to our customers. You will work on ambiguous and complex business and research science problems with large opportunities.You are an individual with outstanding analytical abilities and comfortable working with cross-functional teams and systems.Key Responsibilities:· Interact with business and software teams to understand their business requirements and operational processes· Frame business problems into scalable solutions· Adapt existing and invent new techniques for solutions· Create and track accuracy and performance metrics· Gather data required for analysis and model building· Prototype models by using high-level modeling languages such as R or in software languages such as Python. A software team will be working with you to transform prototypes into production.· Interview, mentor, and develop scientists and engineers across the organization· Create, enhance, and maintain technical documentationWe're a small, independent lab inside AWS working on tooling to improve the operational efficiency across the most popular AWS services. As a lab, we manage an existing portfolio of tools and actively build new services to solve problems.Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.We are dedicated to supporting our new team members. Our team has a broad mix of experience levels and Amazon tenures, and we’re building an environment that celebrates knowledge sharing and mentorship.Our team also puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work.This position requires that the candidate selected be a US Citizen.Key job responsibilitiesOur Data Scientists are scientific generalists who leverage tools from a variety of analytical disciplines to answer business questions and support business decision making. They analyze business questions by defining relevant metrics, proposing proper models, and performing sensitivity analysis. They work with real data in an efficient manner, usually manifested by mastery of scripting languages.Data Scientists work with a variety of quantitative and mathematical methods, understand the tradeoffs among different methods in their most proximate subfield (e.g., Statistics, Operation Research, or Economics) for modeling and experimental design.A day in the lifeHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.About the teamWe are dedicated to supporting our new team members. Our team has a broad mix of experience levels and Amazon tenures, and we’re building an environment that celebrates knowledge sharing and mentorship.Our team also puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work.
US, Virtual
Job summaryAmazon’s Automated Inventory Management (AIM) team is looking for passionate, hard-working, and talented individuals to join our fast paced, stimulating environment to help invent the future of business ownership with Technology, and to translate big data into actionable insights.The AIM team is part of the Supply Chain Optimization Technology (SCOT) Team within the Operations Organization.As an Applied Scientist on the AIM team, you will design quantitative systems, forecasting models and solve real world problems using the latest machine learning techniques. You will also work with a team of Product Managers, Business Intelligence Engineers and Software Engineers to research and build solutions to provide insights to business leaders at the most senior levels throughout the company.Key responsibilities:· Implement statistical and machine learning methods to solve complex business problems· Research new ways to improve predictive and explanatory models· Directly contribute to the design and development of automated prediction systems and ML infrastructure· Build models that can detect supply chain defects and explain variance to the optimal state· Collaborate with other researchers, software developers, and business leaders to define the scientific roadmap for this team
ES, B, Barcelona
Job summaryAre you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon delivery network. Current research and areas of work within our team include machine learning forecast, anomaly detection models, model interpretability, graph neural nets, among others.We are looking for a Manager, Applied Science (Machine Learning) with a strong academic and industry background in the areas of probabilistic machine learning, statistics, and/or time series forecasting.At Amazon, we strive to be the most customer-centric company on earth. For the latter we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
ES, B, Barcelona
Job summaryAre you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon delivery network. Current research and areas of work within our team include machine learning forecast, anomaly detection models, model interpretability, graph neural nets, among others.We are looking for a Senior Machine Learning Scientist with a strong academic background in the areas of machine learning, time series forecasting, and/or anomaly detection.At Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
US, WA, Virtual Location - Washington
Job summaryVoice-driven AI experiences are finally becoming a reality and Amazon’s Alexa voice cloud service and Echo devices are at the forefront of this latest technology wave. We deliver world-class products on aggressive schedules that are used every day, by people you know, in and about their homes. At the same time, we obsess about customer trust and ensure that we build products in a manner that maintains our high bar for customer privacy. We are looking for a passionate and talented Applied Scientist with experience in delivering production systems based on innovative research. This is a unique opportunity to play a key role in an exciting, fast growing business. You will be working on one of the world's most cutting edge customer experience and technology. You'll design and run experiments, research new algorithms, and find new ways of optimizing customer experience. Besides theoretical analysis and innovation, you will work closely with talented engineers and ML scientists to put your algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally.You should thrive in ambiguous environments that require to find solutions to problems that have not been solved before. You enjoy and succeed in fast paced environments where learning new concepts quickly is a must. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience building large-scale distributed systems to creating reliable, scalable, and high performance products. Your strong communication skills enable you to work effectively with both business and technical partners.You will be joining a select group of people making history producing one of the most highly rated products in Amazon's history. Candidates can work in Arlington, VA OR Seattle, WA.
US, WA, Virtual Location - Washington
Job summaryVoice-driven AI experiences are finally becoming a reality and Amazon’s Alexa voice cloud service and Echo devices are at the forefront of this latest technology wave. We deliver world-class products on aggressive schedules that are used every day, by people you know, in and about their homes. At the same time, we obsess about customer trust and ensure that we build products in a manner that maintains our high bar for customer privacy. We are looking for a passionate and talented Applied Scientist with experience in delivering production systems based on innovative research. This is a unique opportunity to play a key role in an exciting, fast growing business. You will be working on one of the world's most cutting edge customer experience and technology. You'll design and run experiments, research new algorithms, and find new ways of optimizing customer experience. Besides theoretical analysis and innovation, you will work closely with talented engineers and ML scientists to put your algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally.You should thrive in ambiguous environments that require to find solutions to problems that have not been solved before. You enjoy and succeed in fast paced environments where learning new concepts quickly is a must. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience building large-scale distributed systems to creating reliable, scalable, and high performance products. Your strong communication skills enable you to work effectively with both business and technical partners.You will be joining a select group of people making history producing one of the most highly rated products in Amazon's history. Candidates can work in Arlington, VA OR Seattle, WA.
US, MA, Boston
Job summaryAre you interested in leading a team to build scalable, state-of-the-art software systems that help make Alexa smarter and more conversational? Do you enjoy working with teams of scientists and engineers to develop novel solutions to tough technical challenges?Amazon is looking for an Applied Science Manager (ASM) to tackle some of the most interesting problems on the leading edge of information retrieval (IR) and machine learning (ML) with our Alexa Artificial Intelligence (AI) team. Alexa AI aims to reinvent information retrieval for a voice-forward, multi-modal future. We enable customers to interact with unstructured and semi-structured content via a broad range of customer experiences including question answering, summarization, search, and multi-turn dialogues.Key job responsibilities* Hire and develop the best. You will help grow and develop a team of scientists and engineers, growing their careers and empowering them to achieve amazing things.* Set a vision for your team and create product roadmaps. Help your team identify technical and product requirements and opportunities. Use project management and prioritization skills to balance long and short-term goals and deliver on your product roadmap.* Keep your technical skills current to contribute to architecture and design discussions. Lead your team to evolve and scale our service architecture.* Drive continual improvements: Set and iteratively improve team processes and mechanisms to make sure the right things get prioritized and work gets done efficiently.A day in the lifeA mix of helping the team architect our systems for massive scale, defining and structuring our science roadmap and establishing the bar for scientific rigor across the team, driving efficiency in our software development processes, career coaching and mentorship, collaborating with sister teams, and interviewing.About the teamOur team is a mix of applied scientists and engineers building a scalable, state-of-the-art information retrieval system supporting several of Alexa’s capabilities. Our members are geographically distributed across the east and west coast, but we’re highly collaborative.
US, CA, Sunnyvale
Job summaryDo you have a passion for diving deep to uncover key insights that drive critical business decisions? If yes, the Customer Behavior Cross Channel Optimization (CBA-XO) team is looking for somebody with your enthusiasm and skills to work as part of the team. We are looking for an experienced data scientistwho is curious, driven, and passionate about marketing insights and analytics. If you are looking for a role where you can make a major impact, we want to meet you.The Customer Behavior Analytics (CBA) organization owns Amazon’s insights pipeline, from data collection to deep analytics. We aspire to be the place where Amazon teams come for answers, a trusted source for data and insights that empower our systems and business leaders to make better decisions. Our outputs shape Amazon product and marketing teams’ decisions and thus how Amazon customers see, use, and value their experience. CBA-XO’s mission is to make Amazon’s marketing the most measurably effective in the world. Our long-term objective is to measure the incremental impact of all Amazon’s marketing investments on consumer perceptions, actions, and sales. This requires measuring Amazon’s marketing comparably and consistently across channels, business teams and countries using a comprehensive approach that integrates all Paid, Owned and Earned marketing activity. As the experts on marketing performance we will lead the Amazon worldwide marketing community by providing critical cross-country insights that can power marketing best practices and tenets globally.This role will work closely with scientists and engineers to develop and run statistical models to understand customer behavior and how customers respond to Amazon’s marketing. You will collaborate directly with economists to produce modeling solutions, partner with software developers and data engineers to build end-to-end data pipelines and production code, and have exposure to senior leadership as we communicate results and provide scientific guidance to the business. You will analyze large amounts of business data, automate, and scale the analysis, and develop metrics that will enable us to continually delight our customers worldwide.Key job responsibilitiesAs a successful scientist, you are an analytical problem solver who enjoys diving into data, is excited about investigations and algorithms, write production code, can multi-task, and can credibly interface between technical teams and business stakeholders. Your analytical abilities, business understanding, and technical savvy will be used to identify specific and actionable opportunities to solve existing business problems and look around corners for future opportunities.Responsibilities Include:Build models and tools using technical knowledge in machine learning, statistical modeling, probability and decision theory, and other quantitative techniques.Implement model in production and develop efficient engineering systems for high throughputUnderstand the business reality behind large sets of data and develop meaningful analytic solutions.Innovate by adapting to new modeling techniques and procedures.Utilizing code (Python, R, etc.) for analyzing data and building statistical models to solve specific business problemsImprove upon existing methodologies by developing new data sources, testing model enhancements, and fine-tuning model parametersCollaborate with researchers, software developers, and business leaders to define product requirements and provide analytical supportCommunicating verbally and in writing to business customers and leadership team with various levels of technical knowledge, educating them about our systems, as well as sharing insights and recommendations
US, WA, Seattle
Job summaryAlexa is the groundbreaking intelligent agent that has changed customers’ lives and made voice driven experiences a reality today. If you are passionate about innovating in the space of artificial intelligence and are a highly-motivated, talented software engineer who wants to bring AI into the lives of our youngest customers, then come join the newly formed Alexa Kids Experience team. To realize our vision of making Alexa a fun, supportive, and educational companion for kids, engineers on our team must innovate to build unique Alexa experiences from ground-up and solve a range of technical challenges across the Alexa architecture. We will leverage speech recognition, natural language processing, interactive voice dialogs, distributed systems, knowledge graph, and much more, to build solutions that scale to millions of devices and customers across the globe.If you are holding out for an opportunity to:· Make a huge impact as an individual· Be part of a team of smart and passionate professionals who will challenge you to grow every day· Solve difficult challenges using your expertise in coding elegant and practical solutions· Create applications at a massive scale used by millions of people· Work with machine learning systems to deliver real experiences, not just researchAnd you are experienced with…· Drive applied science (machine learning) projects end-to-end ~ from ideation, analysis, prototyping, development, metrics, and monitoring· Conduct deep analyses on massive user and contextual data sets· Propose viable modeling ideas to advance optimization or efficiency, with supporting argument, data, or, preferably, preliminary results· Design, develop, and maintain scalable, Machine Learning models with automated training, validation, monitoring and reporting· Stay familiar with the field and apply state-of-the-art Machine Learning techniques to NLP and related optimization problems· Produce peer-reviewed scientific paper in top journals and conferencesAnd you constantly look for opportunities to…· Innovate, simplify, reduce waste, and increase efficiencies· Use data to make decisions and validate assumptions· Automate processes otherwise performed by humansLearn from others and help grow those around you...then we would love to chat!In 2022, we have the opportunity to build new products and features from the ground up and we are looking for strong, bias for action engineering leaders who are not afraid of taking bold bets and trying new things to improve customer experience for Alexa.As part of a new and growing team, you will be iterating on new features and products to help drive innovation and expansion. You will work on cross-functional and cross-domain opportunities; tackle challenging projects aim to accelerate experimentations in Alexa; and build out operating mechanisms and technology to enable novel customer experiences. You will be instrumental in setting the team culture, quality bar, engineering best practices, and norms. Mentoring and growing the team around you will be one of the primary ways you measure your own success. You will have the opportunity to contribute and develop deep expertise in the areas of distributed systems, machine learning, conversational technologies, user interfaces (including voice and natural user interfaces), data storage and data pipelines.This role is exciting for scientists who love to apply startup mindset to their day-to-day, enjoy working cross-functionally to master both business and technology knowledge, and are passionate about building engineering best practices. If you are looking for opportunity to learn, grow and lead, this is the position for you.