“How robust r u?": Evaluating task-oriented dialogue systems on spoken conversations

2021
Download
Abstract
Most prior work in dialogue modeling has been on written conversations mostly because of existing data sets. However, written dialogues are not sufficient to fully capture the nature of spoken conversations as well as the potential speech recognition errors in practical spoken dialogue systems. This work presents a new benchmark on spoken task-oriented conversations, which is intended to study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling. We report that the existing state-of-the-art models trained on written conversations are not performing well on our spoken data, as expected. Furthermore, we observe improvements in task performances when leveraging n-best speech recognition hypotheses such as by combining predictions based on individual hypotheses. Our data set enables speech-based benchmarking of task-oriented dialogue systems.
Download

US, WA, Seattle
Job summaryWork at the intersection of data science and economics.The DAC AdsEcon Team is looking for a Data Scientist II to help and be part of a team to put cutting edge economic and data science advertising research into production. We are looking for a unique individual to help us build a prototype that will have a profound impact in our advertising businesses.Advertising is used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team sits in the Business/Corporate Development, and our charter is to use econometrics, machine learning, and data science to build disruptive products that move the needle in our multiple Amazon Advertising businesses. We also generate insights to guide Amazon Advertising strategy, providing direct support to the high level leaders.If you have a background in economics, computer science, statistics, or mathematics and have a passion for solving large, and impactful problems, this is the job for you. Key responsibilities of Data Scientist include the following:· Partnering with economists and senior team members to drive science improvements and implement technical solutions at the cutting edge of machine learning and econometrics· Helping build data systems that leverage diverse data sources to understand how different advertiser’s decisions impact their performance across multiple advertising products.· Build interpretable statistical models and analyze experiment results to answer questions that will drive high impact decisions across Amazon.About Amazon's Advertising business:Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.
US, NJ, Newark
Job summaryGood storytelling starts with great listening. At Audible, that means each role and every project has our audience in mind. Because the same people who design, develop, and deploy our products also happen to use them. To us, that speaks volumes.ABOUT THIS ROLEAudible is searching for an exceptional data scientist to join our economics team and drive the development of models at the intersection of machine learning and econometrics at scale. The Audible economics organization works across the business to measure and maximize the value Audible delivers to customers, creators, and communities globally. In this role, there will be a focus on partnering with our content and product teams to build a groundbreaking catalog of audiobooks and spoken-word entertainment, develop innovative tools to generate value for creators, and optimize content distribution and monetization.We are looking for someone experienced in building ML models at scale for complex prediction and optimization problems, who also has a background (or burgeoning interest!) in causal inference or interpretable machine learning. In addition to working with our staff economists and data scientists, you will also collaborate closely with scientists across Audible and partner teams at Amazon on problems pertinent to subscription businesses and the production of original media content.As a Data Scientist, you will...· Work with leadership in our content and product organizations to identify key analytical problems and opportunities – your work is expected to be a key input to our future content strategy.· Develop and maintain scalable, innovative data science and machine learning models that deliver actionable insights and results.· Collaborate with other data scientists, economists, and analysts at Audible to build data-driven solutions to key business problems.
US, NJ, Newark
Job summaryGood storytelling starts with great listening. At Audible, that means each role and every project has our audience in mind. Because the same people who design, develop, and deploy our products also happen to use them. To us, that speaks volumes.ABOUT THIS ROLEAudible seeks a Data Scientist who will help our marketing team improve paid marketing efficiency and performance. In this role, you will make the best of your skillset in modeling and general analytics. Modelling: use your knowledge of (un-) supervised learning, reinforcement learning, and simulation to explain, quantify, predict and prescribe. Analytics: use your knowledge of marketing and paid media to translate business and financial goals into insights and influence action. Overall: you will seek to create value for both stakeholders and customers and will convey results in a clear, actionable way to managers and senior leaders.As a Data Scientist, you will...· Will build analytical products end-to-end (decks, dashboards, data science models, simulations) at scale and at speed, from ideation and data extraction to presenting results to stakeholders (from manager to VP level).· Support development of models to optimize the Who, When, Where and How of all our conversations with customers and specifically to measure and optimize paid media.· Develop, maintain, and iterate on Amazon-scale data engineering and modelling pipelines.· Imagine and invent before the business asks, and create groundbreaking applications using cutting-edge approaches.· Contribute to the growth of the Audible Global Insights and Data Science team by sharing your ideas, intellectual property and learning from others.· Work closely with Audible stakeholders to drive the business forward, and deliver impactful models and analyses based on robust economic, financial, and statistical analysis.
US, MA, North Reading
Job summaryAre you an MS or PhD student interested in Robotics, Manipulation, Computer Vision, or Machine Learning? Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?At Amazon Robotics, we strive to push boundaries in order to provide the best possible experience for our customers. We are looking for scientists striving to use their domain expertise to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. As an Applied Scientist intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.As an Applied Scientist intern, you will work from concept through to execution. This role will give you the opportunity to build tools and support structures needed to analyze data, dive deep to resolve root cause of systems errors and changes, and present findings to business partners to drive improvements.Come build the future with us. Amazon internships are full-time (40 hours/week) for 12 or more consecutive weeks with start dates between May and June 2022.Amazon Robotics intern opportunities will be based in the Greater Boston Area, in our two state-of-the-art facilities in Westborough and North Reading, MA. Both campuses provide a unique opportunity for co-ops to have direct access to robotics testing labs and manufacturing facilities!
US, WA, Seattle
Job summaryAmazon’s Shipping and Delivery Support (SDS) team is a part of Amazon World Wide Customer Service dedicated to support successful package deliveries to Amazon Customers. As a Data Scientist on our team, you’ll use Amazon’s wealth of data to help answer tough questions like where and when preemptively intervening with a problem is most likely to result in a successful delivery, which signals should alert us that a delivery is at risk of missing its estimate, and what is the relative value of a specific set of support associate actions as they relate to delivery success. You will also leverage Amazon's rich datasets and machine learning techniques to understand customer urgency, and build algorithms to recommend treatment actions to optimize delivery outcome. This role will be a key member of the Shipping and Delivery Support Science Team.The Senior Data Scientist will work closely with Business Intelligence Engineers, Data Engineers, Product Managers, Software Engineers, and Program Managers to develop statistical and machinelearning models, design and run experiments, and find new ways to improve support experience to optimize the customer experience and Amazon’s on-time deliveries. The Scientist will collaborate with technology and product leaders to solve business and technology problems using scientific approaches to build new services that surprise and delight Amazon drivers and our customers. Science at Amazon is a highly experimental activity, although theoretical analysis and innovation are also welcome. Our scientists work closely with software engineers to put algorithms into practice. They also work on cross-disciplinary efforts with other scientists within Amazon.The key strategic objectives for this role include:· Understanding drivers, impacts, and key influences on delivery success and support contacts.· Optimizing support processes to improve the Customer experience and Amazon’s on time delivery.· Automating feedback loops for algorithms in production.· Collaborate with researchers, software developers, and business leaders to define product requirements and provide analytical support.· Utilizing Amazon systems and tools to effectively work with terabytes of data.· Communicating verbally and in writing to business customers and leadership team with various levels of technical knowledge, educating them about our systems, as well as sharing insights and recommendations
US, NY, New York
Job summaryCalling all inventors! Are you excited about Advertising technology? Love to work at the intersection between Machine Learning, Customer Experience, and Revenue Growth? Keep reading.Amazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our products are strategically important to our businesses driving long term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day!As a Senior Data Scientist for Amazon Business Advertising, you will help build a new Business-to-Business (B2B) advertising experience from the ground up and create and launch features for both advertisers and business shoppers globally. Our team owns end-to-end advertising experience including placements, ad relevance, creative, ad serving, advertiser experience, and marketing. You will work on complex science, engineering, optimization, econometric, and user-experience problems in order to deliver relevant Amazon Business ads on Amazon search and detail pages world-wide. Leveraging Amazon's massive data repository, you will develop experiments, insights and optimizations that enable the monetization of Amazon online and mobile search properties while enhancing the experience of Amazon shoppers.As a Senior Data Scientist on this team you will:* Lead full life-cycle Data Science solutions from beginning to end.* Deliver with independence on challenging large-scale problems with complexity and ambiguity.* Write code (Python, R, Scala, SQL, etc.) to obtain, manipulate, and analyze data to better understand Amazon Business customer shopping journey so as to help enhance their experience at Amazon.* Build Machine Learning and statistical models to solve business problem you define* Conduct written and verbal presentations to share insights to audiences of varying levels of technical sophistication.*Why you will love this opportunity*: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate.*Impact and Career Growth:* You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.Team video ~ https://youtu.be/zD_6Lzw8raEAbout the teamAmazon is building a world class advertising business and defining and delivering a collection of self-service performance advertising products that drive discovery and sales of merchandise. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products.Amazon Business Advertising team's mandate is to build a new Business-to-Business (B2B) advertising experience from the ground up as it requires a differentiated customer and advertiser experience. We are investing in a deep science and technical team to pursue a transformation opportunity.#adptjobs#adptvs#vertcat#amazonbusinessads
US, WA, Seattle
Job summaryAmazon brings buyers and sellers together. Our retail customers depend on us to give them access to every product at the best possible price. Our sellers depend on us to give them a platform to launch their business into every home and marketplace. Making this happen is the mission of every engineer in Amazon's North America Consumer (NAC) organization.To this end, the Science team is tasked with:· Organizing available data sources, and creating detailed dictionaries of data that can be used in future analyses.· Partnering with product teams in evaluating the financial and operational impact of new product offerings.· Conducting research into optimization and machine learning algorithms which can be applied to solve business problems.· Partnering with other scientists in evaluating algorithms and suggestions from a business view point.· Carrying out independent data-backed initiatives that can be leveraged later on in the fields of network organization, costing and financial modeling of processes.In order to execute the above mandate we are on the look out for smart and qualified Data Scientists who will own projects in partnership with product and research teams as well as operate autonomously on independent initiatives that are expected to unlock benefits in the future. A past background in Statistics is necessary, along with advanced proficiency in languages such as Python and R.Key job responsibilitiesAs a Data Scientist, you are able to use a range of advanced analytical methodologies to solve challenging business problems when the solution is unclear. You have a combination of business acumen, broad knowledge of statistics, deep understanding of ML algorithms, and an analytical mindset. You thrive in a collaborative environment, and are passionate about learning. Our team utilizes a variety of AWS tools such as Redshift, Sagemaker, Lambda, S3, and EC2 with a variety of skillsets in Linear and Discrete Optimization, ML, NLP, Forecasting, Probabilistic ML and Causal ML. You will bring knowledge in many of these domains along with your own specialties and skillsets.
US, CA, Pasadena
Job summaryThe Amazon Web Services (AWS) Center for Quantum Computing in Pasadena, CA, is hiring a Quantum Research Scientist to join a multi-disciplinary, fast-paced team of theoretical and experimental physicists, materials scientists, and hardware and software engineers pushing the forefront of quantum computing. The candidate should demonstrate a thorough knowledge of experimental measurement techniques as well as quantum mechanics theory.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.Key job responsibilities* Contribute to fast-paced and agile research to help close the many orders of magnitude gap in gate error rates required for fault tolerant quantum computation* Design and perform experiments to characterize quantum devices in close collaboration with software and engineering teams* Develop models to understand and improve device performance* Effectively document results and communicate to a broad audience* Create robust software for implementation, automation, and analysis of measurements* Specify technical requirements in a cross-team collaboration using analytical arguments derived from physics theoryA day in the life* Analyze experimental data* Develop software to test and run new experiments on existing devices; collaborate with software engineers to achieve high code standard* Debug test setups to achieve high-quality data* Present results and cross-collaborate with others’ work* Perform code review for a colleague’s merge request
US, CA, Pasadena
Job summaryThe Amazon Web Services (AWS) Center for Quantum Computing in Pasadena, CA, is looking to hire a Quantum Research Scientist in the Test and Measurement group. You will join a multi-disciplinary team of theoretical and experimental physicists, materials scientists, and hardware and software engineers working at the forefront of quantum computing. You should have a deep and broad knowledge of experimental measurement techniques.Candidates with a track record of original scientific contributions will be preferred. We are looking for candidates with strong engineering principles, resourcefulness and a bias for action, superior problem solving, and excellent communication skills. Working effectively within a team environment is essential. As a research scientist you will be expected to work on new ideas and stay abreast of the field of experimental quantum computation.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.Key job responsibilitiesIn this role, you will drive improvements in qubit performance by characterizing the impact of environmental and material noise on qubit dynamics. This will require designing experiments to assess the role of specific noise sources, ensuring the collection of statistically significant data, analyzing the results, and preparing clear summaries for the team. Finally, you will work with hardware engineers, material scientists, and circuit designers to implement changes which mitigate the impact of the most significant noise sources.
US, MA, Cambridge
Job summaryThe Alexa Artificial Intelligence (AI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background, to help build industry-leading Speech and Language technology.Key job responsibilitiesAs an Applied Scientist with the Alexa AI team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding.About the teamThe Alexa AI team has a mission to push the envelope in Natural Language Understanding (NLU). Specifically, we focus on incremental learning, continual learning and fairness, in order to provide the best-possible experience for our customers.