We combine diverse research areas spanning from Computer Vision and Robotics to Language Grounding and Neuroscience. Our agent “Symbiote” draws its name from our main vision of SimBot: a mutually beneficial symbiosis between the agent and the human, where the agent assists the human to complete household chores, but at the same time the human teaches the agent to perform progressively harder tasks more efficiently. Symbiote learns to communicate, explore and solve problems in a continual fashion, to quickly adapt to the user’s everyday life.
Faculty advisor: Katerina Fragkiadaki
Nikolaos G. — Team leader
Nikolaos is a second-year Robotics PhD student at Carnegie Mellon University, working with Prof. Katerina Fragkiadaki. He is working on teaching robots common sense, specifically on training agents that continually learn to perceive, imagine, interact and solve visual tasks guided by natural language and 3D world priors. His most recent work, currently under review, employs natural-language guidance to detect harder objects in 2D and 3D scenes, even when state-of-the-art detectors fail. Before joining CMU, he spent 3 years in the industry, first developing a natural-language-understanding commercial chatbot and later performing research on Scene Graph Generation.
Ayush J.
Ayush is a first-year masters student in the Robotics Institute at Carnegie Mellon University working with Prof. Katerina Fragkiadaki. He is working on building embodied assistive robots that are trustworthy, display commonsense and learn continually in a self-supervised way. Recently, he and his team developed a language grounding model that can ground language and find objects mentioned by a human in realistic 2D and 3D scenes. His past research works include enabling an embodied agent to learn about objects without ground truth supervision in an unseen 3D environment by allowing the agent to move around in the scene.
Nilay P.
Nilay is a first-year Masters of Science in Computer Science student in the School of Computer Science at Carnegie Mellon University. He is working with Prof. Katerina Fragkiadaki as a Graduate Student Researcher. Before joining CMU, he worked as a Software Engineer at Google in the Apps Search Quality Team where he worked on improving underlying ML ranking models for Gmail ASO search. Previously, he worked on the joint problem of clustering and generative sampling in a semi-supervised way. He has also worked on Human-Object Interaction, Action recognition in videos, Program Code Generation and Human Pose Estimation in the past.
Gabriel S.
Gabe is a second-year Ph.D. student at Carnegie Mellon University in the joint program between Neural Computation and Machine Learning under the supervision of Dr. Mike Tarr and Dr. Katerina Fragkiadaki. His work is supported by the National Science Foundation Graduate Research Fellowship. Previously, he received a B.S. in Biomedical Engineering from the University of Rochester, where he studied the marmoset visual system under Dr. Jude Mitchell.
Paul S.
Paul is a 3rd-year PhD student in a joint programme between the Institute for Systems and Robotics at Instituto Superior Técnico and the Machine Learning Department at Carnegie Mellon University, working with Prof. José Santos-Victor and Prof. Katerina Fragkiadaki. He is working on learning to represent task relevant contextual information in a sample efficient manner with the overarching goal of applying it to problems such as action prediction and planning in human-AI collaboration scenarios. More recently he had a research internship at Facebook Reality Labs which enabled him to work on context sensitive AI assistants in the AR domain.
Zhou X.
Zhou is a fourth-year PhD student in the Robotics Institute at Carnegie Mellon University, working with Prof. Katerina Fragkiadaki. His research mainly focuses on enabling robots with better manipulation skills via imitation learning, scene representation learning and model-based reinforcement learning. His recent work includes leveraging 3D scene representation for more accurate and generalizable dynamics modeling in robotic manipulation tasks and proposing novel neural architectures that enable faster adaptation of learned dynamics models to novel environments. He is currently looking into how to leverage the power of large-scale language model in manipulation tasks.
Katerina Fragkiadaki — Faculty advisor
Katerina is an assistant professor in the Machine Learning Department at Carnegie Mellon. Prior to joining MLD's faculty, she spent three wonderful years as a post doctoral researcher first at UC Berkeley working with Jitendra Malik and then at Google Research in Mountain View working with the video group. She completed her Ph.D. in GRASP, UPenn with Jianbo Shi.
