ScottyBot

The ScottyBot team hails from Carnegie Mellon University (CMU), and is a joint venture between the Language Technologies and Robotics Institutes.

The CMU School of Computer Science (SCS) is considered to be one of the leading centers of artificial intelligence research in the world, with numerous federal grants, affiliated research institutes, degree programs, and awards in the areas of robotics, language technologies, and human-machine interaction.

simbot.png

Jonathan F. — Team leader

Jonathan is a PhD Candidate in the Language Technologies Institute at CMU and a Research Scientist at Bosch Research. His research focuses on harnessing domain knowledge for multimodal representation learning, in robotics and autonomous driving. As a former researcher in a major U.S. defense contractor and research committee member for various U.S. Department of Energy programs in distributed sensing and control, he brings over a decade's worth of experience in institutional research and advanced development from public, private, and academic sectors. Jonathan holds Bachelor's and Master's degrees in Electrical & Computer Engineering from Carnegie Mellon.

Adhokshaja M.

Adhokshaja is a Masters student at the Language Technologies Institute at Carnegie Mellon University. He is interested in the confluence of the fields of Computer Vision, Multimodal Machine Learning and Reinforcement Learning, currently researching the domain of audio, video and action with Prof. Yonatan Bisk.

Benny J.

Benny is a student in Master of Computational Data Science. He is interested in natural language generation, ML infrastructure and system design. Prior to CMU, he completed his Bachelors in Computer Science from UC Berkeley.

Jessica Z.

Jessica Zhong is a student from Master of Computational Data Science in Languages Technology Institute at CMU. She has a keen interest in computer vision and multimodal machine learning, and she enjoyed solving real-world challenges during Simbot.

Jimin S.

Jimin is a 2nd year Master’s student in Language Technologies at Carnegie Mellon University, where she is co-advised by Yonatan Bisk and Jean Oh. Her research interest is in language grounding and embodied dialog.

Kushagra M.

Kushagra is a graduate student at CMU pursuing a Masters in Computational Data Science. He is advised by Prof. Louis-Philippe Morency and is presently working on Computer Vision problems for AR/VR glasses. His general research interests are in Computer Vision, Deep Learning and Multimodal ML.

Malaika V.

Malaika is a second year masters student in Computational Data Science at Carnegie Mellon University's Language Technologies Institute. She is interested in Natural Language Processing and Computer Vision, and enjoys working on problems in multimodal machine learning

Nikhil G.

Nikhil is a 2nd year Masters student in the Language Technologies Institute at Carnegie Mellon University. His interests lie in NLP and big data analytics. Prior to joining CMU, he was part of the Cloud team at VMWare.

Prasoon V.

Prasoon is a 2nd year Masters student in Computational Data Science at the Language Technologies Institute at CMU. His interests lie in embodied dialogue agents, multimodal representation learning, and safe and responsible AI. Prior to CMU, he worked at the Franchise Analytics group at Goldman Sachs, and completed Bachelors in Computer Science from IIT Varanasi, India.

Sai Vishwas P.

Sai is a Master's student in the Computational Data Science program at CMU. Sai's research interests are in the areas of multimodal machine learning and embodied AI.

Shubham V.

Shubham is a 2nd year Masters student in the Language Technologies Institute at Carnegie Mellon University. His research interests lie in question answering and cloud computing. Prior to joining CMU, he was an app developer at Oracle, and completed his Bachelors in Computer Science from IIT Roorkee, India.

Shubham P.

Shubham is a 2nd year Masters student in the Language Technologies Institute at Carnegie Mellon University. His research interests lie in NLP and multimodal machine learning. Prior to joining CMU, he worked in the Equities Trading group at Morgan Stanley.

Vineeth R.

Vineeth is a second year Masters Student in Language Technologies Institute at Carnegie Mellon University. Vineeth’s research currently focuses on computer vision and multimodal machine learning. Previously, Vineeth worked in the self-driving domain to build large scale perception models for object detection and lane segmentation.

Xinyue C.

Xinyue is a Masters in Computational Data Science student with experience in natural language tasks, including natural language QA and generation.

Yonatan Bisk — Faculty advisor

Yonatan Bisk is an assistant professor in the Languages Technology Institute at CMU. His work broadly falls into uncovering the latent structures of natural language; modeling the semantics of the physical world; and connecting language to perception and control.

Latest news

The latest updates, stories, and more about Alexa Prize.
US, MA, Boston
As a Principal Scientist within the Artificial General Intelligence (AGI) organization, you are a trusted part of the technical leadership. You bring business and industry context to science and technology decisions. You set the standard for scientific excellence and make decisions that affect the way we build and integrate algorithms. You solicit differing views across the organization and are willing to change your mind as you learn more. Your artifacts are exemplary and often used as reference across organization. You are a hands-on scientific leader. Your solutions are exemplary in terms of algorithm design, clarity, model structure, efficiency, and extensibility. You tackle intrinsically hard problems, acquiring expertise as needed. You decompose complex problems into straightforward solutions. You amplify your impact by leading scientific reviews within your organization or at your location. You scrutinize and review experimental design, modeling, verification and other research procedures. You probe assumptions, illuminate pitfalls, and foster shared understanding. You align teams toward coherent strategies. You educate, keeping the scientific community up to date on advanced techniques, state of the art approaches, the latest technologies, and trends. You help managers guide the career growth of other scientists by mentoring and play a significant role in hiring and developing scientists and leads. You will play a critical role in driving the development of Generative AI (GenAI) technologies that can handle Amazon-scale use cases and have a significant impact on our customers' experiences. Key job responsibilities You will be responsible for defining key research directions, adopting or inventing new machine learning techniques, conducting rigorous experiments, publishing results, and ensuring that research is translated into practice. You will develop long-term strategies, persuade teams to adopt those strategies, propose goals and deliver on them. You will also participate in organizational planning, hiring, mentorship and leadership development. You will be technically fearless and with a passion for building scalable science and engineering solutions. You will serve as a key scientific resource in full-cycle development (conception, design, implementation, testing to documentation, delivery, and maintenance). About the team The AGI team has a mission to push the envelope with multimodal LLMs and Gen AI in Computer Vision, in order to provide the best-possible experience for our customers.
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the next level. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
GB, Cambridge
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like Echo, Fire Tablets, Fire TV, and other consumer devices. We are looking for exceptional scientists to join our Applied Science team to advance the state-of-the-art in developing efficient multimodal language models across our product portfolio. Through close hardware-software integration, we design and train models for resource efficiency across the hardware and software tech stack. The Silicon and Solutions Group Edge AI team is looking for a talented Sr. Applied Scientist who will lead our efforts on inventing evaluation methods for multimodal language models and agents for new devices, including audio and vision experiences. Key job responsibilities - Collaborate with cross-functional engineers and scientists to advance the state of the art in multimodal model evaluations for devices, including audio, images, and videos - Invent and validate reliability for novel automated evaluation methods for perception tasks, such as fine-tuned LLM-as-judge - Develop and extend our evaluation framework(s) to support expanding capabilities for multimodal language models - Analyze large offline and online datasets to understand model gaps, develop methods to interpret model failures, and collaborate with training teams to enhance model capabilities for product use cases - Work closely with scientists, compiler engineers, data collection, and product teams to advance evaluation methods - Mentor less experienced Applied Scientists A day in the life As a Scientist with the Silicon and Solutions Group Edge AI team, you'll contribute to innovative methods for evaluating new product experiences and discover ways to enhance our model capabilities and enrich our customer experiences. You'll research new methods for reliably assessing perception capabilities for audio-visual tasks in multimodal language models, design and implement new metrics, and develop our evaluation framework. You'll collaborate across teams of engineers and scientists to identify and root cause issues in models and their system integration to continuously enhance the end-to-end experience. About the team Our Edge AI science team brings together our unique skills and experiences to deliver state-of-the-art multimodal AI models that enable new experiences on Amazon devices. We work at the intersection of hardware, software, and science to build models designed for our custom silicon.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong deep learning background, to help build industry-leading technology with generative AI (GenAI) and multi-modal systems. Key job responsibilities As an Applied Scientist with the AGI team, you will work with talented peers to develop algorithms and modeling techniques to advance the state of the art with multi-modal systems. Your work will directly impact our customers in the form of products and services that make use of vision and language technology. You will leverage Amazon’s large-scale computing resources to accelerate development with multi-modal Large Language Models (LLMs) and GenAI in Computer Vision. About the team The AGI team has a mission to push the envelope with multimodal LLMs and GenAI in Computer Vision, in order to provide the best-possible experience for our customers.
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the limits. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. As an Applied Scientist on our team, you will focus on building state-of-the-art ML models for healthcare. Our team rewards curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the forefront of both academic and applied research in this product area, you have the opportunity to work together with a diverse and talented team of scientists, engineers, and product managers and collaborate with other teams. This role offers a unique opportunity to work on projects that could fundamentally transform healthcare outcomes. Key job responsibilities In this role, you will: • Design and implement novel AI/ML solutions for complex healthcare challenges • Drive advancements in machine learning and data science • Balance theoretical knowledge with practical implementation • Work closely with customers and partners to understand their requirements • Navigate ambiguity and create clarity in early-stage product development • Collaborate with cross-functional teams while fostering innovation in a collaborative work environment to deliver impactful solutions • Establish best practices for ML experimentation, evaluation, development and deployment • Partner with leadership to define roadmap and strategic initiatives You’ll need a strong background in AI/ML, proven leadership skills, and the ability to translate complex concepts into actionable plans. You’ll also need to effectively translate research findings into practical solutions. A day in the life You will solve real-world problems by getting and analyzing large amounts of data, generate insights and opportunities, design simulations and experiments, and develop statistical and ML models. The team is driven by business needs, which requires collaboration with other Scientists, Engineers, and Product Managers across the Special Projects organization. You will prepare written and verbal presentations to share insights to audiences of varying levels of technical sophistication. About the team We represent Amazon's ambitious vision to solve the world's most pressing challenges. We are exploring new approaches to enhance research practices in the healthcare space, leveraging Amazon's scale and technological expertise. We operate with the agility of a startup while backed by Amazon's resources and operational excellence. We're looking for builders who are excited about working on ambitious, undefined problems and are comfortable with ambiguity.
US, WA, Seattle
The Sponsored Products and Brands (SPB) team at Amazon Ads is re-imagining the advertising landscape through state-of-the-art generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond! Key job responsibilities This role will be pivotal in redesigning how ads contribute to a personalized, relevant, and inspirational shopping experience, with the customer value proposition at the forefront. Key responsibilities include, but are not limited to: - Contribute to the design and development of GenAI, deep learning, multi-objective optimization and/or reinforcement learning empowered solutions to transform ad retrieval, auctions, whole-page relevance, and/or bespoke shopping experiences. - Collaborate cross-functionally with other scientists, engineers, and product managers to bring scalable, production-ready science solutions to life. - Stay abreast of industry trends in GenAI, LLMs, and related disciplines, bringing fresh and innovative concepts, ideas, and prototypes to the organization. - Contribute to the enhancement of team’s scientific and technical rigor by identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. - Mentor and grow junior scientists and engineers, cultivating a high-performing, collaborative, and intellectually curious team. A day in the life As an Applied Scientist on the Sponsored Products and Brands Off-Search team, you will contribute to the development in Generative AI (GenAI) and Large Language Models (LLMs) to revolutionize our advertising flow, backend optimization, and frontend shopping experiences. This is a rare opportunity to redefine how ads are retrieved, allocated, and/or experienced—elevating them into personalized, contextually aware, and inspiring components of the customer journey. You will have the opportunity to fundamentally transform areas such as ad retrieval, ad allocation, whole-page relevance, and differentiated recommendations through the lens of GenAI. By building novel generative models grounded in both Amazon’s rich data and the world’s collective knowledge, your work will shape how customers engage with ads, discover products, and make purchasing decisions. If you are passionate about applying frontier AI to real-world problems with massive scale and impact, this is your opportunity to define the next chapter of advertising science. About the team The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond!
US, MA, Boston
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As an Applied Scientist with the AGI team, you will work with talented peers to lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
GB, Cambridge
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like Echo, Fire Tablets, Fire TV, and other consumer devices. We are looking for exceptional scientists to join our Applied Science team to advance the state-of-the-art in developing efficient multimodal language models across our product portfolio. Through close hardware-software integration, we design and train models for resource efficiency across the hardware and software tech stack. The Silicon and Solutions Group Edge AI team is looking for a talented Applied Scientist who will develop new evaluation methods for multimodal language models and agents for our devices, including audio and vision experiences. Key job responsibilities - Collaborate with cross-functional engineers and scientists to advance the state of the art in multimodal model evaluations for devices, including audio, images, and videos - Invent and validate reliability for novel automated evaluation methods for perception tasks, such as fine-tuned LLM-as-judge - Develop and extend our evaluation framework(s) to support expanding capabilities for multimodal language models - Analyze large offline and online datasets to understand model gaps, develop methods to interpret model failures, and collaborate with training teams to enhance model capabilities for product use cases - Work closely with scientists, compiler engineers, data collection, and product teams to advance evaluation methods A day in the life As a Scientist with the Silicon and Solutions Group Edge AI team, you'll contribute to innovative methods for evaluating new product experiences and discover ways to enhance our model capabilities and enrich our customer experiences. You'll have opportunities to collaborate across teams of engineers and scientists to bring algorithms and models to production. About the team Our Edge AI science team brings together our unique skills and experiences to deliver state-of-the-art multimodal AI models that enable new experiences on Amazon devices.
US, WA, Seattle
The Sponsored Products and Brands (SPB) team at Amazon Ads is re-imagining the advertising landscape through state-of-the-art generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond! Key job responsibilities This role will be pivotal in redesigning how ads contribute to a personalized, relevant, and inspirational shopping experience, with the customer value proposition at the forefront. Key responsibilities include, but are not limited to: - Contribute to the design and development of GenAI, deep learning, multi-objective optimization and/or reinforcement learning empowered solutions to transform ad retrieval, auctions, whole-page relevance, and/or bespoke shopping experiences. - Collaborate cross-functionally with other scientists, engineers, and product managers to bring scalable, production-ready science solutions to life. - Stay abreast of industry trends in GenAI, LLMs, and related disciplines, bringing fresh and innovative concepts, ideas, and prototypes to the organization. - Contribute to the enhancement of team’s scientific and technical rigor by identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. - Mentor and grow junior scientists and engineers, cultivating a high-performing, collaborative, and intellectually curious team. A day in the life As an Applied Scientist on the Sponsored Products and Brands Off-Search team, you will contribute to the development in Generative AI (GenAI) and Large Language Models (LLMs) to revolutionize our advertising flow, backend optimization, and frontend shopping experiences. This is a rare opportunity to redefine how ads are retrieved, allocated, and/or experienced—elevating them into personalized, contextually aware, and inspiring components of the customer journey. You will have the opportunity to fundamentally transform areas such as ad retrieval, ad allocation, whole-page relevance, and differentiated recommendations through the lens of GenAI. By building novel generative models grounded in both Amazon’s rich data and the world’s collective knowledge, your work will shape how customers engage with ads, discover products, and make purchasing decisions. If you are passionate about applying frontier AI to real-world problems with massive scale and impact, this is your opportunity to define the next chapter of advertising science. About the team The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value.
US, WA, Seattle
The Sponsored Products and Brands (SPB) team at Amazon Ads is re-imagining the advertising landscape through state-of-the-art generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. Key job responsibilities This role will be pivotal in redesigning how ads contribute to a personalized, relevant, and inspirational shopping experience, with the customer value proposition at the forefront. Key responsibilities include, but are not limited to: - Contribute to the design and development of GenAI, deep learning, multi-objective optimization and/or reinforcement learning empowered solutions to transform ad retrieval, auctions, whole-page relevance, and/or bespoke shopping experiences. - Collaborate cross-functionally with other scientists, engineers, and product managers to bring scalable, production-ready science solutions to life. - Stay abreast of industry trends in GenAI, LLMs, and related disciplines, bringing fresh and innovative concepts, ideas, and prototypes to the organization. - Contribute to the enhancement of team’s scientific and technical rigor by identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. - Mentor and grow junior scientists and engineers, cultivating a high-performing, collaborative, and intellectually curious team. A day in the life As an Applied Scientist on the Sponsored Products and Brands Off-Search team, you will contribute to the development in Generative AI (GenAI) and Large Language Models (LLMs) to revolutionize our advertising flow, backend optimization, and frontend shopping experiences. This is a rare opportunity to redefine how ads are retrieved, allocated, and/or experienced—elevating them into personalized, contextually aware, and inspiring components of the customer journey. You will have the opportunity to fundamentally transform areas such as ad retrieval, ad allocation, whole-page relevance, and differentiated recommendations through the lens of GenAI. By building novel generative models grounded in both Amazon’s rich data and the world’s collective knowledge, your work will shape how customers engage with ads, discover products, and make purchasing decisions. If you are passionate about applying frontier AI to real-world problems with massive scale and impact, this is your opportunity to define the next chapter of advertising science. About the team The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond!