nFlux team photo
nFlux team members (left to right) Aditya Mukewar, Chenlei Zhang, Karthik Ramkumar, Adam Phillips, Falisha Kanji, Seyed Sajjadi*, Anton Safarevich*,  Natan Vargas,  Pulin Agrawal, Collin Miller*,  Danny Pena* (* indicates co-founder).
Credit: nFlux

3 questions with Seyed Sajjadi: How to utilize a video analytics platform to automate the process of learning

Sajjadi, a co-founder and CEO of Alexa Fund company nFlux.ai, explains how procedure monitoring can help humans, from astronauts to manufacturers, and even home cooks.

Editor’s Note: This interview is the latest installment within a series Amazon Science is publishing related to the science behind products and services from companies in which Amazon has invested. In 2019, the Alexa Fund first invested in nflux.ai, and then in 2020 participated in the company’s seed round.

In 2018, Seyed Sajjadi was pursuing a master’s degree in computer science at the University of Southern California (USC) when he decided to drop out and found nFlux.ai. While pursuing his master’s degree, he also was working as a project manager at the Systems Engineering Research Laboratory (SERL) research laboratory at California State University in Northridge, Calif.

At the University of Southern California, Seyed Sajjadi focused on the development of Sigma, a cognitive architecture and system. One outcome of the research was this paper, which Sajjadi coauthored with computer science professor Paul Rosenbloom and other USC collaborators.

At USC, Sajjadi was working as a member of the Cognitive/Virtual Human Architecture lab under computer science professor Paul Rosenbloom. There, he focused on the development of Sigma, a cognitive architecture and system that strives to combine what has been learned from four decades of independent work on symbolic cognitive architectures, probabilistic graphic models, and more recently neural models. One outcome of his research there was a paper, “Controlling Synthetic Characters in Simulation: A Case for Cognitive Architectures and Sigma”, which Sajjadi coauthored with Rosenbloom and other USC collaborators. The paper was accepted to the 2018 Interservice/Industry Training Simulation and Education Conference (I/ITSEC).

At SERL, Sajjadi led an interdisciplinary team of more than 90 engineers and human factors researchers focused on building the next generation of robotic search-and-rescue systems with artificial intelligence. It was here that Sajjadi and colleagues began thinking about forming nflux.ai, inspired, he says, by the fictional character J.A.R.V.I.S. (Just A Rather Very Intelligent System) from the Marvel Cinematic Universe film franchise, and a vision for how artificial intelligence systems can augment humans in positive ways.

Amazon Science asked Sajjadi three questions about the challenges of developing cognitive architectures, nFlux’s focus on imitation learning within the manufacturing sector, and how the company’s technology could eventually be relevant to Alexa customers at home.

Q. What is a video analytics platform, and how does it enable what you call procedure monitoring?

nFlux is the first intelligent video analytics platform that automates the process of learning and generating contextual insights from the unstructured data inside video footage. One of our goals is to pass a Turing test for video comprehension. Imagine there is a woman sitting at a desk looking at a video on her computer. We want to develop a video comprehension system that can answer any question about that video with the same level of comprehension as the woman.

Our first customer was NASA, and right now we’re working to build a system similar to HAL 9000, the fictional AI character in the Space Odyssey series. HAL 9000 is a general AI system that can mimic the way humans think, behave, and take actions. Ironically, Space Odyssey is centered around a deep-space mission. Today, if astronauts have a question, they call Houston, and someone at Johnson Space Center answers their questions. But as we embark on deep space missions, such as Mars, where there is a 40-minute delay in communication, that method of communication isn’t practical. So we want to provide an intelligent system on the spacecraft that can understand what the astronauts are doing and assist them by augmenting what they’re capable of doing on their own.

Seyed Sajjidi
Seyed Sajjadi

That’s what we refer to as procedure monitoring, which is the core of the innovation we’re developing. Our objective is imitation learning, or learning by demonstration. If an astronaut is performing a procedure, our objective is to capture that procedure via video with a minimum number of examples, say 10 or 15, which in machine learning is a tiny sample size. But from that small sample size we develop a computational model so that if another astronaut has to perform that same procedure in the future, we can track that. If in performing that procedure the astronaut deviates from the procedure, perhaps by missing a screw, our system can recognize that in real time and alert the astronaut.

That’s really the core of what we consider procedure monitoring, or the astronaut-assistant technology we’ve been developing. One of the keys to our video analytics platform is its ability to learn from a minimum number of videos. That’s significant.

But for those algorithms to infer from a small set of data, they are extracting basic signals from our base models.   This is possible since the agent can be augmented with prior semantic knowledge of key activities, such as tethering, drilling components, etc., and can recognize key components — objects, tools — of each step from synthetically generated data. This technique is inspired by the way humans ingest information as they watch a new procedure they have never seen before. We are capable of recognizing the key activity being performed even if we have not previously seen the objects/tools being used, and can deduce the steps required to successfully complete a procedure.

Q. How is nFlux technology being applied within the manufacturing sector?

Despite the perception that robots have taken over the manufacturing floor, seventy-two percent of manufacturing work is still done by humans. Six million people here in the United States go to work every day to perform a certain set of procedures. As that person on the manufacturing floor is doing her job, we can capture any deviations in real time.

Our system can be a virtual teacher or instructor helping train a new employee, or an existing employee who’s learning a new procedure. This is extremely valuable to manufacturers because it reduces production cycles. If they can train employees faster at their manufacturing facilities that translates into millions of dollars in manufacturing time. It also impacts the quality of their products. The better a manufacturer’s employees are trained, and the more standardized their procedures, the lower their defect rates. Those are two critical elements to any manufacturer.

Our technology also helps in capturing what we refer to as tribal knowledge.  In many complicated manufacturing environments, training can’t be provided on a piece of paper, instead you need a computational model derived from video of how the procedure is conducted properly. That computational model can help train new employees as they come on board, monitor their work to ensure they’re following procedures properly, and act as that intelligent assistant for your manufacturing workforce. nFlux isn’t designed to replace the workforce, it’s there to augment the work they’re doing. Ultimately, this reduces the amount of rework required to output high-quality products from that manufacturing plant

Q. The Alexa Fund is an investor. So how could your computational model be relevant to Alexa customers?

Echo Show stationed on a kitchen counter.
Imagine, says Sajjadi, that as you were cooking the Echo Show 10 was watching you and could alert you if you missed an ingredient. That, he says, would be an example of taking procuedure monitoring from the shop floor to the kitchen.

An Echo Show with a screen was first introduced in 2017, and since there have been subsequent generations, including the new Echo Show 10, which first became available earlier this year. These devices support multimodal experiences, providing Alexa greater context and an understanding with vision. These multimodal Echo devices tend to be in the kitchen and one of the most popular uses is for cooking, and following cooking instructions in real time. Imagine if as you were cooking the Echo Show 10 was watching you cook and alerted you if you missed adding an ingredient. That would be an example of taking procedure monitoring from the shop floor to the kitchen. 

Earlier this year, we were awarded another NASA contract to support the health of astronauts. This work is relevant to other Alexa healthcare-related scenarios. If you’re an elderly person living at home or within an assisted living facility, what if an nFlux application noticed that you didn’t take your pills at 9 a.m. as you are supposed to, and alerted you. Or what if you’re under your doctor’s orders to walk for five minutes every two hours. We could recognize that you haven’t been mobile in the past couple of hours, and remind you to walk. These are the kinds of consumer-facing scenarios that complement our commercial approach to procedure monitoring, and could be applied in the home. 

Research areas

Related content

US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies to deliver game changing value to our customers. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon’s Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon’s on-line retail business. As an economist on our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies with the potential to deliver game changing value to our customers. This is an opportunity for a high-energy individual to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals.
US, CA, San Francisco
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of statistical inference, experimentation design, economic theory and machine learning to design new methods and pricing strategies for assessing pricing innovations. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon's Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon's on-line retail business. As an economist on our team, you will will have the opportunity to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in experimentation design, applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of interns from previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US
The Amazon Supply Chain Optimization Technology (SCOT) organization is looking for an Intern in Economics to work on exciting and challenging problems related to Amazon's worldwide inventory planning. SCOT provides unique opportunities to both create and see the direct impact of your work on billions of dollars’ worth of inventory, in one of the world’s most advanced supply chains, and at massive scale. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Knowledge of econometrics and Stata/R/or Python is necessary, and experience with SQL, Hadoop, and Spark would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Seattle
The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. We are looking for an Intern Economist with excellent coding skills to design and develop rigorous models to assess the causal impact of fees on third party sellers’ behavior and business performance. As a Science Intern, you will have access to large datasets with billions of transactions and will translate ambiguous fee related business problems into rigorous scientific models. You will work on real world problems which will help to inform strategic direction and have the opportunity to make an impact for both Amazon and our Selling Partners.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, Hadoop, Spark and Python would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking interns and co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it! Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, artificial intelligence, human-robot interaction, optimization and more.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.