New method improves knowledge-graph-based question answering

Replacing hand annotation with a machine learning component reduces labor, while an intersection operation enables multiple-entity queries.

Question answering is a popular task in natural-language processing, where models are given questions such as “What city is the Mona Lisa in?” and trained to predict correct answers, such as “Paris”. 

One way to train question-answering models is to use a knowledge graph, which stores facts about the world in a structured format. Historically, knowledge-graph-based question-answering systems required separate semantic-parsing and entity resolution models, which were costly to train and maintain. But since 2020, the field has been moving toward differentiable knowledge graphs, which allow a single end-to-end model to take questions as inputs and directly output answers.

At this year’s Conference on Empirical Methods in Natural Language Processing (EMNLP), we presented two extensions to end-to-end question answering using differentiable knowledge graphs.

In “End-to-end entity resolution and question answering using differentiable knowledge graphs”, we explain how to integrate entity resolution into a knowledge-graph-based question-answering model, so that it will be performed automatically. This is a potentially labor-saving innovation, as some existing approaches require hand annotation of entities.

And in “Expanding end-to-end question answering on differentiable knowledge graphs with intersection”, we explain how to handle queries that involve multiple entities. In experiments involving two different datasets, our approach improved performance on multi-entity queries by 14% and 19%.

Traditional approaches

In a typical knowledge graph, the nodes represent entities, and the edges between nodes represent relationships between entities. For example, a knowledge graph could contain the fact “Mona Lisa | exhibited at | Louvre Museum”, which links the entities “Mona Lisa” and “Louvre Museum” with the relationship “exhibited at”. Similarly, the graph could link the entities “Louvre Museum” and “Paris” with the relationship “located in”. A model can learn to use the facts in the knowledge graph to get to answer questions.

GraphTraversal_Cropped.gif
Question answering through traversal of links in a knowledge graph.

Traditional approaches to knowledge-graph-based question answering use a pipeline of models. First, the question goes into a semantic-parsing model, which is trained to predict queries. Queries can be thought of as instructions of what to do in the knowledge graph — for example, “find out where the ‘Mona Lisa’ is exhibited, then look up what city it’s in.” 

Next, the question goes into an entity resolution model, which links parts of the sentence, like “Mona Lisa”, to IDs in the knowledge graph, like Q12418.

PipelinedApproach.gif
The pipelined approach to knowledge-graph-based question answering.

While this pipeline approach works, it does have some flaws. Each model is trained independently, so it has to be evaluated and updated separately, and each model requires annotations for training, which are time-consuming and expensive to collect. 

End-to-end question answering

End-to-end question answering is a way to rectify these flaws. During training, an end-to-end question answering model is given the question and the answer but no instructions about what to do in the knowledge graph. Instead, the model learns the instructions based on the correct answer. 

In 2020, Cohen et al. proposed a way to perform end-to-end question answering using differentiable knowledge graphs, which represent knowledge graphs as tensors and queries as differentiable mathematical operations. This allows for fully differentiable training, so that the answer alone provides a training signal to update all parts of the model.

DifferentiableApproach.gif
End-to-end question answering with a differentiable knowledge graph.

“End-to-end entity resolution and question answering using differentiable knowledge graphs”

In our first paper, we extend end-to-end models to include the training of an entity resolution component. Previous work left the task of entity resolution (linking “Mona Lisa” to Q12418) out of the scope of the model, relying instead on a separate entity resolution model or hand annotation of entities. 

We propose a way to train entity resolution as part of the question-answering model. To do this, we start with a baseline model similar to the implementation by Cohen et al., which has an encoder-decoder structure and an attention mechanism that takes in a question and returns predicted answers with probabilities.

We add entity resolution to this baseline model by first introducing a span detection component. This identifies all the possible parts of the sentence (spans) that could refer to an entity. For example, in the question “Who wrote films starring Tom Hanks?”, there are multiple spans, such as “films” or “Tom”, and we want the model to learn to give a higher score to the correct span, “Tom Hanks”. 

Entity resolution.png
An end-to-end model that jointly learns knowledge-graph-based question answering and entity resolution.

Then for each of the identified spans, our model ranks all the possible entities in the knowledge graph that the span could refer to. For example, “Tom Hanks” is also the name of a seismologist and a theologian, but we want the model to learn to give the actor a higher score.

Span detection and entity resolution happen jointly in a new entity resolution component, which returns possible entities with scores. Finding the entities in the knowledge graph and following the paths from the inference component yields the predicted answers. We believe that this joint modeling is responsible for our method’s increase in efficiency.

In our experiments, we use two English question-answering datasets. We find that although our ER and E2E models perform slightly worse than the baseline, they do come very close, with differences of about 7% and 5% on the two datasets. 

This is an impressive result, since the baseline model uses hand-annotated entities from the datasets, while our model is learning entity resolution jointly with question answering. With these findings, we demonstrate the feasibility of learning to perform entity resolution and multihop inference in a single end-to-end model.

“Expanding end-to-end question answering on differentiable knowledge graphs with intersection”

In our second paper, we extend end-to-end models to handle more complex questions with multiple entities. Take for example the question “Who did Natalie Portman play in Star Wars?”. This question has two entities, Natalie Portman and Star Wars. 

Previous end-to-end models were trained to follow paths originating with one entity in a knowledge graph. However, this is not enough to answer questions with multiple entities. If we started at “Natalie Portman” and found all the roles she played, we would get roles that were not in Star Wars. If we started at Star Wars and found all the characters, we would get characters that Natalie Portman didn’t play. What we need is an intersection of the characters played by Natalie Portman and the characters in Star Wars. 

To handle questions with multiple entities, we expand an end-to-end model with a new operation: intersection. For each entity in the question, the model follows paths from the entity independently and arrives at an intermediate answer. Then the model performs intersection, which we implemented as the element-wise minimum of two vectors, to identify which entities the intermediate answers have in common. Only entities that appear in all intermediate answers are returned in the final answer. 

Multiple entities.png
An end-to-end knowledge-graph-based question-answering model for queries with multiple entities.

In our experiments, we use two English question-answering datasets. Our results show that introducing intersection improves performance over the baseline by 3.7% on one and 8.9% on the other. 

More importantly, we see that improved performance comes from better handling of questions with multiple entities, where the intersection model surpasses the baseline by over 14% on one dataset and by 19% on the other.

In future work, we plan to continue developing end-to-end models by improving entity resolution, so it’s competitive with hand annotation of entities; integrating entity resolution with intersection; and learning to handle more-complex operations, such as maximums/minimums and counts.

Amazon at EMNLP

Read more about Amazon's presence at EMNLP, including papers, membership in the organizing committee, and workshop and tutorial involvement.

Related content

US, WA, Bellevue
Amazon Fulfillment Planning & Execution (FPX) Science team within Supply Chain Optimization Technologies (SCOT) Fulfilment Optimization group is seeking a Principal Research Scientist with expertise in Machine Learning and a proven record of solving business problems through scalable ML solutions. Network Planning and Fulfillment Execution tackles some of the most mathematically complex challenges in facility and transportation planning to improve Amazon's operational efficiency worldwide. We own Amazon’s global fulfillment center and transportation topology planning and execution. The team also owns the short-term network planning that determines the optimal flow of customer orders through Amazon fulfillment network. This includes developing sophisticated math models and controllers that assign orders to fulfillment centers to be picked and packed and then planning the optimal ship method in terms of cost, speed and carbon impact to deliver to the customer. These plans drive downstream decisions that are in the billions of dollars. The systems we build are entirely in-house, and are on the cutting edge of both academic and applied research in large scale supply chain planning, optimization, machine learning and statistics. These systems operate at various scales, from real-time decision system that completes thousands of transactions per seconds, to large scale distributed system that optimize Amazon’s fulfillment network. As Amazon continues to build and expand the first party delivery network, this role will be critical to realize this vision. Your tech solution will have large impacts to the physical supply chain of Amazon, and play a key role in improving Amazon consumer business’s long-term profitability. If you are interested in diving into a multi-discipline, high impact space this is the team for you. Key job responsibilities As a Principal Research Scientist within FPX Science team, you will propose and deploy solutions that will likely draw from a range of scientific areas such as supervised, semi-supervised and unsupervised learning, reinforcement learning, advanced statistical modeling, and graph models. You will have an opportunity to be on the forefront of supply chain thought leadership by working on some of the most difficult problems in the industry, with some of the best product managers, research scientists, statisticians, and software engineers to integrate scientific work into production systems. You will partner with the senior tech leaders in the organization to define the long-term vision of our Network Planning and Fulfillment Execution systems. You will play a key role in developing long term strategic solutions that have business impact beyond the scope of the organization. You will bring deep technical expertise in the area of Machine Learning, and will play an integral part in building Amazon's Fulfillment Optimization systems. Other responsibilities include: • Research and develop machine learning models to solve diverse business problems faced within Network Planning and Fulfillment Execution team. • Drive and execute machine learning projects/products end-to-end: from ideation, analysis, prototyping, development, metrics, and monitoring. • Review and audit modeling processes and results for other scientists, both junior and senior. • Advocate the right ML solutions to business stakeholders, engineering teams, as well as executive level decision makers • You will ensure senior leaders in the organization are up to speed on important trends, tools and technologies and how they will be used to impact the business. A day in the life In this role, you will be a technical leader in machine learning with significant scope, impact, and high visibility. Your solutions will impact business segments worth many-billions-of-dollars and geographies spanning multiple countries and markets. As a Principal Research Scientist on the team, you will be involved in every aspect of the process - from ideation, business analysis and scientific research, through to development and deployment of advanced models - giving you a real sense of ownership. From day one, you will be working with bar raising scientists, engineers, and designers. You are expected to make decisions about technology, models and methodology choices. You will also collaborate with the broader science community in Amazon to broaden the horizon of your work and mentor engineers and other scientists. We are seeking someone who wants to lead projects that require innovative thinking and deep technical problem-solving skills to create production-ready machine learning solutions. A successful candidate is able to quickly approach large ambiguous problems, turn high-level business requirements into mathematical models, identify the right solution approach, and contribute to the software development for production systems. Successful candidates must thrive in fast-paced environments, which encourage collaborative and creative problem solving, be able to measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs. We look for individuals who know how to deliver results and show a desire to develop themselves, their colleagues, and their career. About the team FPX Science team contains a group of scientists with different technical backgrounds including Machine Learning and Operations Research, who will collaborate closely with you on your projects. Our team directly supports multiple functional areas across Fulfillment Optimization and the research needs of the corresponding product and engineering teams. We tackle some of the most mathematically complex challenges in facility and transportation planning to improve Amazon's operational efficiency worldwide and at a scale that is unique to Amazon. We often seek the opportunity of applying hybrid techniques in the space of Operations Research and Machine Learning to tackle some of our biggest technical challenges. We disambiguate complex supply chain problems and create ML and optimization solutions to solve those problems at scale. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Seattle
Amazon Shipping and Delivery Support (SDS) Tech team is seeking a passionate and customer-obsessed Senior Data Scientist to join our science team. You will use scientific research and rigorous analytics to influence our program and product strategies in driver and recipient support, solve complex problems at large scale, and drive intelligence and innovation in decision making. In this role, your main focus is to perform analysis, synthesize information, identify business opportunities, provide project direction, and communicate design and technical requirements within the team and across stakeholder groups. You will assist in defining trade-offs and quantifying opportunities for a variety of projects. You will learn current processes, build metrics, educate diverse stakeholder groups, assist product and tech teams in initial solution design, and audit new process flow implementations. Key job responsibilities * Provide thought leadership and support the development of continuously-evolving business analytics and data models, own the quantitative analysis of project opportunity and ROI. * Translate difficult business problem statements into data science frameworks; build, evaluate, and optimize statistical and machine learning models to solve focused business problems. * Retrieve, analyze, and synthesize critical data into a format that is immediately useful to answering specific questions or informing operational decisions. * Collaborate with product, program, and operations teams to design experiments (A/B Test) and analyze results to support launch decisions. * Conduct written and verbal presentations to share insights to audiences of varying levels of technical sophistication. A day in the life If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA | Seattle, WA, USA
US, VA, Arlington
We are seeking a Data Scientist to join our analytics team. This person will own the design and implementation of scalable and reliable approaches to support or automate decision making throughout the business. You will do this by analyzing data with a variety of statistical techniques and then building, validating, and implementing models based your analysis. You will not be able to do this alone but by building partnerships across data, engineering, and business teams. Key job responsibilities - Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult customer or business problems and cases in which the solution approach is unclear. - Proactively seek to identify business opportunities and insights and provide solutions to automate and optimize key internal and external products based on a broad and deep knowledge of Amazon data, industry best-practices, and work done by other teams. - Dive deep into the data and other models across the business to identify defects or inefficiencies which materially impact the customer or business, but can be mitigated through corrective actions for the AB Ops use case - Acquire this data by accessing data sources and building the necessary SQL/ETL queries or scripts. - Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. - Build models and automated tools using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. - Validate these models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. - Implement these models in a manner which complies with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production. - Enable product engineering teams to consume your models through services which can directly power customer-facing experiences. - Inspect the key business metrics/KPIs (even if you did not create them) when your analytics work points to potential gaps or opportunities; providing clear, compelling analyses by leveraging your knowledge across the AWS suite of products to support the broader business. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Seattle, WA, USA
GB, London
Have you ever ordered a product on Amazon and when that box with the smile arrived you wondered how it got to you so fast? Have you wondered where it came from and how much it cost Amazon to deliver it to you? We are looking for a Senior Data Scientist who will be responsible to develop cutting-edge scientific solutions to optimize our Pan-European fulfillment strategy, to maximize our Customer Experience and minimize our cost and carbon footprint. You will partner with the worldwide scientific community to help design the optimal fulfillment strategy for Amazon. You will also collaborate with technical teams to develop optimization tools for network flow planning and execution systems. Finally, you will also work with business and operational stakeholders to influence their strategy and gather inputs to solve problems. To be successful in the role, you will need deep analytical skills and a strong scientific background. The role also requires excellent communication skills, and an ability to influence across business functions at different levels. You will work in a fast-paced environment that requires you to be detail-oriented and comfortable in working with technical, business and technical teams. Key job responsibilities - Design and develop mathematical models to optimize inventory placement and product flows. - Design and develop statistical and optimization models for planning Supply Chain under uncertainty. - Manage several, high impact projects simultaneously. - Consult and collaborate with business and technical stakeholders across multiple teams to define new opportunities to optimize our Supply Chain. - Communicate data-driven insights and recommendations to diverse senior stakeholders through technical and/or business papers. We are open to hiring candidates to work out of one of the following locations: London, GBR
GB, London
Re-imagining the realms of what’s possible in advertising. Amazon is re-imagining advertising. Amazon Ads operates at the intersection of eCommerce and advertising and offering a rich array of advertising solutions and audience insights so businesses and brands can create relevant campaigns that produce measurable results. At Amazon Ads, you can build models that impact millions every day. And we’re passionate about solving real-world problems while using cutting-edge machine learning and artificial intelligence to do this. For example, our applied science teams leverage a variety of advanced machine learning and cloud computing techniques to power Amazon's advertising offerings. This includes building algorithms and cloud services using clustering, deep neural networks, and other ML approaches to make ads more relevant while respecting privacy. They develop machine learning models to predict ad outcomes and select the optimal ad for each shopper, context, and advertiser objective, leveraging techniques like multi-task learning, bandit/reinforcement learning, counterfactual estimation, and low-latency extreme ML. The teams also utilize Spark, EMR, and Elasticsearch to extract insights from big data and deliver recommendations to advertisers at scale, continuously improving through offline analysis and impact evaluation. Additionally, they apply generative AI models for dynamic creative optimization and video experimentation and automation. Underpinning these efforts are unique technical challenges, such as operating at unprecedented scale (hundreds of thousands of requests per second with 40ms latency) while respecting privacy and customer trust guarantees, and solving a wide variety of complex computational advertising problems related to traffic quality, viewability, brand safety, and more. Help us take innovation in advertising to the next level. Our teams are based in our fast-growing tech hubs in London and Edinburgh. Learn more about Amazon Ads, employee stories and available opportunities here: https://www.amazon.jobs/content/en/teams/advertising/applied-science-machine-learning-research?ref_=a20m_us_car_lp_asml Key job responsibilities * Design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both analysis and business judgment. * Collaborate with software engineering teams to integrate successful experiments into large-scale, highly complex Amazon production systems. * Promote the culture of experimentation and applied science at Amazon. * Demonstrate ability to meet deadlines while managing multiple projects. * Excel communication and presentation skills working with multiple peer groups and different levels of management * Influence and continuously improve a sustainable team culture that exemplifies Amazon’s leadership principles We are open to hiring candidates to work out of one of the following locations: Edinburgh, MLN, GBR | London, GBR
GB, Cambridge
The Artificial General Intelligence team (AGI) has an exciting position for an Applied Scientist with a strong background NLP and Large Language Models to help us develop state-of-the-art conversational systems. As part of this team, you will collaborate with talented scientists and software engineers to enable conversational assistants capabilities to support the use of external tools and sources of information, and develop novel reasoning capabilities to revolutionise the user experience for millions of Alexa customers. Key job responsibilities As an Applied Scientist, you will develop innovative solutions to complex problems to extend the functionalities of conversational assistants . You will use your technical expertise to research and implement novel algorithms and modelling solutions in collaboration with other scientists and engineers. You will analyse customer behaviours and define metrics to enable the identification of actionable insights and measure improvements in customer experience. You will communicate results and insights to both technical and non-technical audiences through written reports, presentations and external publications. We are open to hiring candidates to work out of one of the following locations: Cambridge, GBR | London, GBR
GB, Cambridge
The Artificial General Intelligence team (AGI) has an exciting position for an Applied Scientist with a strong background NLP and Large Language Models to help us develop state-of-the-art conversational systems. As part of this team, you will collaborate with talented scientists and software engineers to enable conversational assistants capabilities to support the use of external tools and sources of information, and develop novel reasoning capabilities to revolutionise the user experience for millions of Alexa customers. Key job responsibilities As an Applied Scientist, you will develop innovative solutions to complex problems to extend the functionalities of conversational assistants . You will use your technical expertise to research and implement novel algorithms and modelling solutions in collaboration with other scientists and engineers. You will analyse customer behaviours and define metrics to enable the identification of actionable insights and measure improvements in customer experience. You will communicate results and insights to both technical and non-technical audiences through written reports, presentations and external publications. We are open to hiring candidates to work out of one of the following locations: Cambridge, GBR | London, GBR
US, VA, Arlington
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The Generative AI Innovation Center at AWS is a new strategic team that helps AWS customers implement Generative AI solutions and realize transformational business opportunities. This is a team of strategists, data scientists, engineers, and solution architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and make plans for launching solutions at scale. The GenAI Innovation Center team provides guidance on best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for Data Scientists capable of using GenAI and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. A key focus of this role is GenAI model customization using techniques such as fine-tuning and continued pre-training to help customers build differentiating solutions with their unique data. Key job responsibilities As a Data Scientist, you will: Collaborate with AI/ML scientists and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms to address real-world challenges Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production Create and deliver best practice recommendations, tutorials, blog posts, sample code, and presentations adapted to technical, business, and executive stakeholder Provide customer and market feedback to Product and Engineering teams to help define product direction About the team Sales, Marketing and Global Services (SMGS) AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest-growing small- and mid-market accounts to enterprise-level customers, including the public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to customer success. AWS Support also partners with a global list of customers that are building mission-critical applications on top of AWS services. The Professional Services team is part of Global Services. About AWS Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Denver, CO, USA | Herndon, VA, USA | New York, NY, USA | Santa Clara, CA, USA | Seattle, WA, USA | Washington Dc, DC, USA
GB, London
Re-imagining the realms of what’s possible in advertising. Amazon is re-imagining advertising. Amazon Ads operates at the intersection of eCommerce and advertising and offering a rich array of advertising solutions and audience insights so businesses and brands can create relevant campaigns that produce measurable results. At Amazon Ads, you can build models that impact millions every day. And we’re passionate about solving real-world problems while using cutting-edge machine learning and artificial intelligence to do this. For example, our applied science teams leverage a variety of advanced machine learning and cloud computing techniques to power Amazon's advertising offerings. This includes building algorithms and cloud services using clustering, deep neural networks, and other ML approaches to make ads more relevant while respecting privacy. They develop machine learning models to predict ad outcomes and select the optimal ad for each shopper, context, and advertiser objective, leveraging techniques like multi-task learning, bandit/reinforcement learning, counterfactual estimation, and low-latency extreme ML. The teams also utilize Spark, EMR, and Elasticsearch to extract insights from big data and deliver recommendations to advertisers at scale, continuously improving through offline analysis and impact evaluation. Additionally, they apply generative AI models for dynamic creative optimization and video experimentation and automation. Underpinning these efforts are unique technical challenges, such as operating at unprecedented scale (hundreds of thousands of requests per second with 40ms latency) while respecting privacy and customer trust guarantees, and solving a wide variety of complex computational advertising problems related to traffic quality, viewability, brand safety, and more. Help us take innovation in advertising to the next level. Our teams are based in our fast-growing tech hubs in London and Edinburgh. Learn more about Amazon Ads, employee stories and available opportunities here: https://www.amazon.jobs/content/en/teams/advertising/applied-science-machine-learning-research?ref_=a20m_us_car_lp_asml Key job responsibilities * Design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both analysis and business judgment. * Collaborate with software engineering teams to integrate successful experiments into large-scale, highly complex Amazon production systems. * Promote the culture of experimentation and applied science at Amazon. * Demonstrate ability to meet deadlines while managing multiple projects. * Excel communication and presentation skills working with multiple peer groups and different levels of management * Influence and continuously improve a sustainable team culture that exemplifies Amazon’s leadership principles * Develop a deep and wide understanding of large ad tech solutions to which you will contribute, and how they interact with components owned by other teams. * Anticipate obstacles and look around corners, effectively prioritising work, solving trade-offs and influencing the development of advertising products beyond the scope of your immediate team. We are open to hiring candidates to work out of one of the following locations: Edinburgh, MLN, GBR | London, GBR
CN, 11, Beijing
Amazon Search JP builds features powering product search on the Amazon JP shopping site and expands the innovations to world wide. As an Applied Scientist on this growing team, you will take on a key role in improving the NLP and ranking capabilities of the Amazon product search service. Our ultimate goal is to help customers find the products they are searching for, and discover new products they would be interested in. We do so by developing NLP components that cover a wide range of languages and systems. As an Applied Scientist for Search JP, you will design, implement and deliver search features on Amazon site, helping millions of customers every day to find quickly what they are looking for. You will propose innovation in NLP and IR to build ML models trained on terabytes of product and traffic data, which are evaluated using both offline metrics as well as online metrics from A/B testing. You will then integrate these models into the production search engine that serves customers, closing the loop through data, modeling, application, and customer feedback. The chosen approaches for model architecture will balance business-defined performance metrics with the needs of millisecond response times. Key job responsibilities - Designing and implementing new features and machine learned models, including the application of state-of-art deep learning to solve search matching, ranking and Search suggestion problems. - Analyzing data and metrics relevant to the search experiences. - Working with teams worldwide on global projects. Your benefits include: - Working on a high-impact, high-visibility product, with your work improving the experience of millions of customers - The opportunity to use (and innovate) state-of-the-art ML methods to solve real-world problems with tangible customer impact - Being part of a growing team where you can influence the team's mission, direction, and how we achieve our goals We are open to hiring candidates to work out of one of the following locations: Beijing, 11, CHN | Shanghai, 31, CHN