Search results

18,776 results found
  • Amazon Scholar
  • Priyanka Sen, Emine Yilmaz
    EMNLP 2020 Workshop on Insights from Negative Results in NLP
    2020
    Collecting training data for semantic parsing is a time-consuming and expensive task. As a result, there is growing interest in industry to reduce the number of annotations required to train a semantic parser, both to cut down on costs and to limit customer data handled by annotators. In this paper, we propose uncertainty and traffic-aware active learning, a novel active learning method that uses model
  • Ting-Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatnia, Pei Zhou, Dilek Hakkani-Tür
    EMNLP 2020 Workshop on DeeLIO
    2020
    Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information
  • Abhinav Aggarwal, Zekun Xu, Oluwaseyi Feyisetan, Nathanael Teissier
    EMNLP 2020 Workshop on PrivateNLP
    2020
    A common metric for assessing the performance of binary classifiers is the Log-Loss score, which is a real number indicating the cross entropy distance between the predicted distribution over the labels and the true distribution (a point distribution defined by the ground truth labels). In this paper, we show that a malicious modeler, upon obtaining access to the Log-Loss scores on its predictions, can exploit
  • Greg Hanneman, Georgiana Dinu
    WMT 2020
    2020
    The ability of machine translation (MT) models to correctly place markup is crucial to generating high-quality translations of formatted input. This paper compares two commonly used methods of representing markup tags and tests the ability of MT models to learn tag placement via training data augmentation. We study the interactions of tag representation, data augmentation size, tag complexity, and language
  • Dustin Axman, Reda Yacouby
    EMNLP 2020 Workshop on Eval4NLP
    2020
    In pursuit of the perfect supervised NLP classifier, razor thin margins and low-resource test-sets can make modeling decisions difficult. Popular metrics such as Accuracy, Precision, and Recall are often insufficient as they fail to give a complete picture of the model’s behavior. We present a probabilistic extension of Precision, Recall, and F1 score, which we refer to as confidence-Precision (cPrecision
  • EMNLP 2020 Workshop on Noisy User-Generated Text (W-NUT)
    2020
    True-casing, the task of restoring proper case to (generally) lower case input, is important in downstream tasks and for screen display. In this paper, we investigate truecasing as an intrinsic task and present several experiments on noisy user queries to a voice-controlled dialog system. In particular, we compare a rule-based, an n-gram language model (LM) and a recurrent neural network (RNN) approaches
  • Applied Scientist
  • Principal, Applied Scientist
  • Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, Nathanael Teissier
    EMNLP 2020 Workshop on PrivateNLP
    2020
    Balancing the privacy-utility tradeoff is a crucial requirement of many practical machine learning systems that deal with sensitive customer data. A popular approach for privacy-preserving text analysis is noise injection, in which text data is first mapped into a continuous embedding space, perturbed by sampling a spherical noise from an appropriate distribution, and then projected back to the discrete
  • Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, JIE MA, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan
    EMNLP 2020
    2020
    In this paper, we propose a neural architecture and a set of training methods for ordering events by predicting temporal relations. Our proposed models receive a pair of events within a span of text as input and they identify temporal relations (Before, After, Equal, Vague) between them. Given that a key challenge with this task is the scarcity of annotated data, our models rely on either pretrained representations
  • Senior Principal Scientist
  • Priyanka Sen, Amir Saffari
    EMNLP 2020
    2020
    While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses
  • Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, Bing Xiang
    EMNLP 2020
    2020
    We propose a generative framework for joint sequence labeling and sentence-level classification. Our model performs multiple sequence labeling tasks at once using a single, shared natural language output space. Unlike prior discriminative methods, our model naturally incorporates label semantics and shares knowledge across tasks. Our framework is general purpose, performing well on few-shot, low-resource
  • Zhiheng Huang, Davis Liang, Peng Xu, Bing Xiang
    Findings of EMNLP 2020
    2020
    Transformer architectures rely on explicit position encodings in order to preserve a notion of word order. In this paper, we argue that existing work does not fully utilize position information. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review absolute position embeddings and existing methods for relative position embeddings. We then propose
  • Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, Bing Xiang
    EMNLP 2020
    2020
    Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document’s language model, were very successful in various IR tasks in the past. However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead. Recently, deep generative
  • Isabelle G. Lee, Vera Zu, Sai Srujana Buddi, Dennis Liang, Purva Kulkarni, Jack G. M. FitzGerald
    EMNLP 2020
    2020
    Virtual Assistants can be quite literal at times. If a user says tell Bob I love him, most virtual assistants will extract the message I love him and send it to the user’s contact named Bob, rather than properly converting the message to I love you. We designed a system that takes a voice message from one user, converts the point of view of the message, and then delivers the result to its target user. We
US, CA, Pasadena
The Amazon Center for Quantum Computing in Pasadena, CA, is looking to hire a Fabrication R&D Scientist with experience in semiconductor process development who will aid in Amazon’s effort to bring cloud quantum computing services to its worldwide customer base. You will join a multi-disciplinary team of scientists, and hardware and software engineers working at the forefront of quantum computing. Through your work inside and outside of the cleanroom environment in the fabrication research and development group, you will solve problems related to developing next-generation quantum processors. Key job responsibilities Candidates must have a demonstrated background in sound scientific and engineering principles, and must have excellent data analysis, bias for action, problem solving, and communication skills, and be highly motivated and curious to research and learn new technical topics as needed. As a Fab R&D scientist you will be expected to work on new ideas and stay abreast of novel approaches in fabricating and packaging superconducting quantum processors. Working effectively within a team environment is critical. A day in the life The candidate will develop novel technologies using micro-/nano-fabrication techniques inside the cleanroom (independently or in collaboration with other scientists and engineers) for next-generation quantum computing. Outside the cleanroom, the candidate will plan experiments, analyze data, and conceive future innovations. About the team Candidates must have a demonstrated background in sound scientific and engineering principles, and must have excellent data analysis, bias for action, problem solving, and communication skills, and be highly motivated and curious to research and learn new technical topics as needed. As a Fab R&D scientist you will be expected to work on new ideas and stay abreast of novel approaches in fabricating and packaging superconducting quantum processors. Working effectively within a team environment is critical. Diverse Experiences Amazon values diverse experiences. Even if you do not meet all the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Export Control Requirement Due to applicable export control laws and regulations, candidates must be either a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum, or be able to obtain a US export license. If you are unsure if you meet these requirements, please apply and Amazon will review your application for eligibility. Key job responsibilities Responsibilities include developing and optimizing processes to fabricate high-coherence superconducting qubits; developing advanced 3DI interconnect and routing technologies for integrating superconducting quantum technologies; analyzing inline metrology and electrical test data; developing and maintaining integration documentation, design rules, and standard operating procedures; interacting with project leads to provide feedback that continuously improves different processes; staying updated with the latest advancements and industry trends in process integration and apply knowledge to improve processes and drive innovation providing technical guidance and support to junior colleagues, fostering a collaborative and knowledge-sharing work environment.
US, NY, New York
Sponsored Products and Brands at Amazon Ads is reimagining the advertising landscape through industry-leading generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of reinventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle, from ad creation and optimization to performance analysis and customer insights. We deliver billions of ad impressions and millions of clicks daily, and are breaking fresh ground to improve both the shopper and advertiser experience. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. The General Shopping Intelligence (GSI) team is a highly motivated, collaborative, and fun-loving group with a strong entrepreneurial spirit and bias for action. We provide advanced real-time machine learning services that connect shoppers with the right ads across all platforms and surfaces worldwide. Through deep understanding of both shoppers and products, we help shoppers discover new products they love, enable advertisers to reach their customers most efficiently, and help Amazon continuously innovate on behalf of all customers. We are seeking a motivated Applied Scientist who loves to innovate at the intersection of customer experience, deep learning, generative AI and high-scale machine learning systems. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. Key job responsibilities As an Applied Scientist, you will: * Conduct deep data analysis to derive insights to the business, and identify gaps and new opportunities * Develop scalable and effective machine-learning models and Generative AI solutions to solve business problems * Run regular A/B experiments, gather data, and perform statistical analysis * Work closely with software engineers to deliver end-to-end solutions into production * Improve the scalability, efficiency and automation of large-scale data analytics, model training, deployment and serving * Conduct research on new generative AI modeling to optimize all aspects of Sponsored Products and Brands business About the team We are pioneers in applying advanced machine learning and generative AI algorithms in Sponsored Products and Brands business. We empower every customer with a customized discovery experiences from back-end optimization (such as customized response prediction models) to front-end CX innovation (such as widgets), to help shoppers feel understood and shop efficiently on and off Amazon.
US, WA, Seattle
Amazon Leo is a constellation of Low Earth Orbit satellites that will provide low-latency, high-speed broadband network connectivity to unserved and underserved communities around the world. We are looking for an Applied Scientist to be a founding scientist on the Engineering and R\&D team within Leo Infrastructure and IP Security. The team defends the manufacturing lines, launch sites, and global ground infrastructure behind the constellation from the most sophisticated threat actors on the planet. These requirements create open scientific problems at the intersection of agentic AI, real-time stream processing, graph-based reasoning, and behavioral analytics. You will build the science behind a neurosymbolic reasoning platform and the models that detect the behavior of sophisticated threat actors. This is an R\&D role with a production mandate, where you define the problem rather than solve a pre-scoped one, and every model, detection, and agent workflow you build becomes the system Leo's security teams use to protect the constellation. #### Export Control Requirement Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. Key job responsibilities - Design and implement scalable, production-grade neurosymbolic systems that integrate symbolic reasoning over graph-based knowledge representations with LLM agents to deliver reliable, verifiable security outcomes. - Design and run reinforcement learning and fine-tuning pipelines (GRPO, PPO, DPO) to optimize language models for security reasoning, triage, and detection-authoring tasks. - Build behavioral and statistical models that detect threat actor behavior, and design the evaluation frameworks that measure model performance against that behavior before trusting a model in production. - Design and build multi-agent systems that autonomously triage, enrich, and contain security events, including the constrained reasoning, safety guardrails, and validation mechanisms that make automated decisions trustworthy at scale. - Own the end-to-end science lifecycle, from research and experimentation through production deployment, defining the metrics that measure system performance and real-world security impact. - Advance the state of the art through publications at top-tier venues, patents, or open-source contributions, and shape the scientific agenda and research culture from day one. A day in the life You will move between research and production in the same week: framing an ambiguous security problem as a scientific question, prototyping an approach, and partnering with software engineers to ship it as a capability the platform runs continuously. Security engineers on your team translate threat intelligence into the adversary behaviors that matter; you build the models that detect those behaviors and evaluate model performance against them. You will obsess over the two latencies that define the platforms, the time from event to detection and the time from detection to containment action, and design agents and detections that drive both down. You will backtest candidate detections against retained telemetry, review evaluation results before a model or agent capability graduates to automated execution, and deliver scientific artifacts that ship. About the team Leo Infrastructure and IP Security protects the people, facilities, hardware, and supply chain behind a global satellite constellation. The Engineering and R\&D team within this organization builds the platforms and tooling the security pillar teams operate on, moving security operations from manual triage to correlation-based detection, automated response, and agentic AI. The team is composed of applied scientists, software engineers, and security engineers working across physical and digital security domains. #### Inclusive Team Culture In Amazon Security, it's in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest security challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. #### Training & Career Growth We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional. #### Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve.
US, NY, New York
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through cutting-edge generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. Key job responsibilities - Define and lead science initiatives from problem framing through production deployment in a high-ambiguity environment - Develop and deploy models spanning computer vision, language, and search and retrieval that operate on multimodal inputs at scale - Design and analyze large-scale online experiments to measure impact on shopper and advertiser outcomes - Collaborate with engineering, product, and design to ship science into production A day in the life As an Applied Scientist on the Sponsored Videos team, you will tackle problems at the intersection of computer vision, generative AI, search and retrieval, and personalization. You'll own the full science lifecycle from research and experimentation through online testing and production deployment, working closely with engineering, product, and design partners to bring ideas to market. You should be comfortable working with multimodal signals, building models that operate at scale, and measuring impact through rigorous experimentation. Your work will directly influence the experience of hundreds of millions of shoppers and the outcomes of tens of thousands of advertisers. About the team The Sponsored Videos team within Sponsored Products and Brands develops the science and systems behind video advertising experiences that connect advertisers and shoppers across Amazon. We are on a mission to make Amazon the best-in-class destination for shoppers to discover, engage with, and build affinity with brands through videos.
US, WA, Bellevue
Are you passionate about shaping the future of Amazon's long-term success with Generative AI and Reinforcement Learning (RL)? We are looking for a strong Applied Scientist to join Long-Term Planning and Forecasting (LTPF) within Supply Chain Optimization Technologies (SCOT). Our mission is to accelerate the Amazon flywheel and ensure Amazon's long-term success by leveraging GenAI and RL to build the forecasting, optimization, and analytics products that plan Amazon's Topline, Inventory, and Fulfillment Capacity and Topology. In this role, you will develop our next-generation forecasting, planning and insight generation products powered by GenAI and reinforcement learning. You will design and implement novel, sophisticated scientific solutions to large-scale problems that directly shape how Amazon plans for the years ahead. You will collaborate closely with applied scientists, research scientists, economists, product managers, and software engineers. We are seeking innovative thinkers who can balance theoretical rigor with practical implementation. Your ability to communicate complex technical concepts and drive data-driven decisions at massive scale will be crucial to your success.
US, WA, Seattle
Data Scientist We are seeking a talented Data Scientist to join our growing science team. In this role, you will leverage your analytical and technical expertise to extract valuable insights from complex datasets, develop machine learning models, and drive data-informed decision-making across the organization. Key Responsibilities: Design and execute end-to-end data science projects, from problem definition to solution deployment Develop and implement machine learning models to solve business problems and improve operational efficiency Create and maintain data pipelines for model training and deployment Conduct exploratory data analysis to uncover patterns and trends Collaborate with stakeholders to understand business requirements and translate them into technical solutions Present findings and recommendations to technical and non-technical audiences Monitor and optimize model performance Key job responsibilities We are looking for talented Data Scientists who are adept at predictive modeling, reinforcement learning, recommendations, and familiarity with LLMs, latent diffusion, or related foundational models that will accelerate our plans to dynamically optimize ad creatives on behalf of advertisers. Every member of the team is expected to build customer (advertiser) facing features, contribute to the collaborative spirit within the team, publish, patent, and raise the research bar within the team. * Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, perform hands-on analysis and modeling of enormous data sets to develop insights that improve shopper experiences and merchandise sales * Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. * Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. * Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. * Research new and innovative machine learning approaches. * Promote the culture of experimentation and data science at Amazon About the team Our team, CreativeX Optimization, is responsible for personalizing the visual experience of ads in real time. We apply advanced machine learning and generative AI to help advertisers build, identify, and understand what ad creatives work with their audiences. We develop customized CTR and CVR prediction models to help advertisers efficiently connect with shoppers across the web. We develop generative models to understand shopper preferences across text, images, and videos. We have a broad mandate to experiment and innovate, as we seek to leapfrog the competition and impact billions of impressions daily.
US, WA, Seattle
Data Scientist We are seeking a talented Data Scientist to join our growing science team. In this role, you will leverage your analytical and technical expertise to extract valuable insights from complex datasets, develop machine learning models, and drive data-informed decision-making across the organization. Key Responsibilities: Design and execute end-to-end data science projects, from problem definition to solution deployment Develop and implement machine learning models to solve business problems and improve operational efficiency Create and maintain data pipelines for model training and deployment Conduct exploratory data analysis to uncover patterns and trends Collaborate with stakeholders to understand business requirements and translate them into technical solutions Present findings and recommendations to technical and non-technical audiences Monitor and optimize model performance Key job responsibilities We are looking for talented Data Scientists who are adept at predictive modeling, reinforcement learning, recommendations, and familiarity with LLMs, latent diffusion, or related foundational models that will accelerate our plans to dynamically optimize ad creatives on behalf of advertisers. Every member of the team is expected to build customer (advertiser) facing features, contribute to the collaborative spirit within the team, publish, patent, and raise the research bar within the team. * Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, perform hands-on analysis and modeling of enormous data sets to develop insights that improve shopper experiences and merchandise sales * Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. * Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. * Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. * Research new and innovative machine learning approaches. * Promote the culture of experimentation and data science at Amazon About the team Our team, CreativeX Optimization, is responsible for personalizing the visual experience of ads in real time. We apply advanced machine learning and generative AI to help advertisers build, identify, and understand what ad creatives work with their audiences. We develop customized CTR and CVR prediction models to help advertisers efficiently connect with shoppers across the web. We develop generative models to understand shopper preferences across text, images, and videos. We have a broad mandate to experiment and innovate, as we seek to leapfrog the competition and impact billions of impressions daily.
CA, BC, Vancouver
Success in any organization begins with its people and having a comprehensive understanding of our workforce and how we best utilize their unique skills and experience is paramount to our future success. WISE (Workforce Intelligence powered by Scientific Engineering) delivers the scientific and engineering foundation that powers Amazon's enterprise-wide workforce planning ecosystem. Addressing the critical need for precise workforce planning, WISE enables a closed-loop mechanism essential for ensuring Amazon has the right workforce composition, organizational structure, and geographical footprint to support long-term business needs with a sustainable cost structure. We are looking for a Sr. Applied Scientist to join our ML/AI team to work on Advanced Optimization and LLM solutions. You will partner with Software Engineers, Machine Learning Engineers, Data Engineers and other Scientists, TPMs, Product Managers and Senior Management to help create world-class solutions. We're looking for people who are passionate about innovating on behalf of customers, demonstrate a high degree of product ownership, and want to have fun while they make history. You will leverage your knowledge in machine learning, advanced analytics, metrics, reporting, and analytic tooling/languages to analyze and translate the data into meaningful insights. You will have end-to-end ownership of operational and technical aspects of the insights you are building for the business, and will play an integral role in strategic decision-making. Further, you will build solutions leveraging advanced analytics that enable stakeholders to manage the business and make effective decisions, partner with internal teams to identify process and system improvement opportunities. As a tech expert, you will be an advocate for compelling user experiences and will demonstrate the value of automation and data-driven planning tools in the People Experience and Technology space. Key job responsibilities * Engineering execution - drive crisp and timely execution of milestones, consider and advise on key design and technology trade-offs with engineering teams * Priority management - manage diverse requests and dependencies from teams * Process improvements – define, implement and continuously improve delivery and operational efficiency * Stakeholder management – interface with and influence your stakeholders, balancing business needs vs. technical constraints and driving clarity in ambiguous situations * Operational Excellence – monitor metrics and program health, anticipate and clear blockers, manage escalations To be successful on this journey, you love having high standards for yourself and everyone you work with, and always look for opportunities to make our services better.
US, WA, Seattle
Join us at the forefront of Amazon's sustainability initiatives to work on environmental and social advancements that support Amazon's long-term worldwide sustainability strategy. At Amazon, we're working to be the most customer-centric company on earth. To get there, we need exceptionally talented, bright, and driven people. We are looking for a Research Scientist to join our growing Sustainability team to drive the science behind value chain decarbonization. This role will establish Amazon's scientific methodologies for sector- and cross-sectoral decarbonization mechanisms, and establish benchmarks for automated validation and risk assessment. As a Research Scientist, you will be responsible for independently leading assessments of environmental issues across the full spectrum of Amazon businesses and evaluating sustainability impacts across the value chain. You will independently develop quality frameworks and methodologies that enable Amazon to scale procurement of high-quality environmental interventions while maintaining scientific rigor and environmental integrity. Key job responsibilities - Develop quality assessment frameworks for complex environmental interventions, baseline-setting approaches, and measurement methodologies - Build quantitative benchmark and statistical models that enable scalable evaluation across heterogeneous data sources - Create attribution methodologies for supply chain interventions across Amazon's diverse footprint - Develop social and environmental safeguard criteria that integrate community impact assessments - Collaborate with cross-functional teams including procurement, sustainability operations, and business units to translate scientific methodologies into operational requirements - Work under the direction of senior business leaders while acting as lead Subject Matter Expert for value chain decarbonization science, including designing and leading research, data collection, modeling, documentation, interpretation, and validation About the team Diverse Experiences: World Wide Sustainability values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Inclusive Team Culture: It’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (inclusive diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth: We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, NY, New York
We are seeking an Research Scientist to lead the development of evaluation frameworks and data collection protocols for robotic capabilities. In this role, you will focus on designing how we measure, stress-test, and improve robot behavior across a wide range of real-world tasks. Your work will play a critical role in shaping how policies are validated and how high-quality datasets are generated to accelerate system performance. You will operate at the intersection of robotics, machine learning, and human-in-the-loop systems, building the infrastructure and methodologies that connect teleoperation, evaluation, and learning. This includes developing evaluation policies, defining task structures, and contributing to operator-facing interfaces that enable scalable and reliable data collection. The ideal candidate is highly experimental, systems-oriented, and comfortable working across software, robotics, and data pipelines, with a strong focus on turning ambiguous capability goals into measurable and actionable evaluation systems. Key job responsibilities - Design and implement evaluation frameworks to measure robot capabilities across structured tasks, edge cases, and real-world scenarios - Develop task definitions, success criteria, and benchmarking methodologies that enable consistent and reproducible evaluation of policies - Create and refine data collection protocols that generate high-quality, task-relevant datasets aligned with model development needs - Build and iterate on teleoperation workflows and operator interfaces to support efficient, reliable, and scalable data collection - Analyze evaluation results and collected data to identify performance gaps, failure modes, and opportunities for targeted data collection - Collaborate with engineering teams to integrate evaluation tooling, logging systems, and data pipelines into the broader robotics stack - Stay current with advances in robotics, evaluation methodologies, and human-in-the-loop learning to continuously improve internal approaches - Lead technical projects from conception through production deployment - Mentor junior scientists and engineers