Jon Tamir_Lab_Photos_0001.jpg
Jon Tamir, an assistant professor of electrical and computer engineering at the University of Texas at Austin, wants to improve how MRI data is acquired. In 2020, he received an Amazon Machine Learning Research Award to support the work.
The University of Texas at Austin

How new machine learning techniques could improve MRI scans

Amazon Research Award recipient Jonathan Tamir is focusing on deriving better images faster.

For many patients, time moves at a glacial pace during a magnetic resonance imaging (MRI) scan. Those who have had one know the challenge of holding impossibly still inside a buzzing, knocking scanner for anywhere from several minutes to more than an hour.

Jonathan (Jon) Tamir is developing machine learning methods to shorten exam times and extract more data from this essential — but often uncomfortable — imaging process.

AWS re:Invent 2022: Impact through cutting-edge ML research with Amazon Research Awards

MRI machines use the body's response to strong magnetic fields and radiofrequency waves to produce pictures of our insides, helping to detect disease and monitor treatments. Just like any image, an MRI scan begins with raw data. Tamir, who is an assistant professor of electrical and computer engineering at the University of Texas at Austin, wants to improve how that data is acquired and derive better images faster. In 2020, he received an Amazon Machine Learning Research Award from Amazon Web Services (AWS) to support the work.

A lack of 'ground-truth' MRI data

Contrary to how the experience might feel to patients inside them, MRI machines move incredibly fast, collecting thousands of measurements at intervals spanning tens or hundreds of milliseconds. The measurements depend on the order and frequency of how magnetic forces and radiofrequency currents are applied to the area being surveyed. Clinicians run specific sequences tailored to the body part and purpose for the MRI.

CT scanner
MRI machines move incredibly fast, collecting thousands of measurements at intervals spanning tens or hundreds of milliseconds. The measurements depend on the order and frequency of how magnetic forces and radiofrequency currents are applied to the area being surveyed. Clinicians run specific sequences tailored to the body part and purpose for the MRI.
Engelstad Photography/Image Supply Co/Adobe

To get the highest possible image quality, an MRI technologist must collect all possible measurements, building from low to high frequency. Each layer of added data results in clearer and more detailed images, but collecting that much data takes far too long. Given the need for expedience, only a subset of the data can be acquired. Which data? "That depends on how we're planning to reconstruct the image," Tamir explained.

At his Computational Sensing and Imaging Lab, Tamir is working with colleagues to optimize both the methods for capturing scans and the image reconstruction algorithms that process the raw information. A key problem: lack of available "ground-truth" data: "That's a very big issue in medical imaging compared to the rest of the machine learning world,” he says.

Related content
Gari Clifford, the chair of the Department of Biomedical Informatics at Emory University and an Amazon Research Award recipient, wants to transform healthcare.

With millions of MRIs generated each year in the United States alone, it might seem surprising that Tamir and colleagues lack data. The final image of an MRI, however, has been post-processed down to a few megabytes. The raw measurements, on the other hand, might amount to hundreds of megabytes or gigabytes that aren't saved by the scanner.

"Different research groups spend a lot of effort building high-quality datasets of ground-truth data so that researchers can use it to train algorithms," Tamir said. "But these datasets are very, very limited."

Another issue, he added, is the fact that many MRIs aren't static images. They are movies of a biological process, such as a heart beating. An MRI scanner is not fast enough to collect fully sampled data in those cases.

Random sampling

Tamir and colleagues are working on machine learning algorithms that can learn from limited data to fill in the blanks, so to speak, on images. One tactic being explored by Tamir and others is to randomly collect about 25% of the possible data from a scan and train a neural network to reconstruct an entire image based on that under-sampled data. Another strategy is to use machine learning to optimize the sampling trajectory in the first place.

Related content
With an encoder-decoder architecture — rather than decoder only — the Alexa Teacher Model excels other large language models on few-shot tasks such as summarization and machine translation.

"Random sampling is a very convenient approach, but we could use machine learning to decide the best sampling trajectory and figure out which points are most important," he said.

In “Robust Compressed Sensing MRI with Deep Generative Priors”, which was presented at the Neural Information Processing Systems (NeurIPS) 2021 conference, Tamir and colleagues at UT-Austin demonstrated a deep learning technique that achieves high-quality image reconstructions based on under-sampled scans from New York University’s fastMRI dataset and the MRIData.org dataset from Stanford University and University of California (UC) Berkeley. Both are publicly available for research and education purposes.

MRI scan stock image
At his Computational Sensing and Imaging Lab, Jon Tamir is working with colleagues to optimize both the methods for capturing scans and the image reconstruction algorithms that process the raw information.
Engelstad Photography/Image Supply Co/Adobe

Other approaches to the problem of image reconstruction have utilized end-to-end supervised learning, which performs well when trained on specific anatomy and measurement models but tends to degrade when faced with the aberrations common in clinical practice.

Instead, Tamir and colleagues used distribution learning, in which a probabilistic model learns to approximate images without reference to measurements. In this case, the model can be used both when the measurement process changes, for example, when changing the sampling trajectory, as well as when the imaging anatomy changes, such as when switching from brain scans to knee scans that the model hasn’t seen before.

'"We're really excited to use this as a base model for tackling these bigger issues we’ve been talking about, such as optimally choosing the measurements to collect, and working with less fully available ground-truth data," Tamir said.

Tamir and his colleagues have published three additional papers related to the Amazon Research Award. One focuses on using hyberbolic geometry to represent data; another uses unrolled alternating optimization to speed MRI reconstruction. Tamir has also developed an open-source simulator for MRI that can be run on GPUs in a distributed way to find the best scan parameters for a specific reconstruction.

The road to clinical adoption

A conventional MRI assembles the image via calculations based on the fast Fourier transform, a bedrock algorithm that resolves combinations of different frequencies. "An inverse fast Fourier transform is all it takes to turn the raw data into an image," he said. "That can happen in less than a few milliseconds. It's very simple."

But in his work with machine learning, Tamir is doing those basic operations in an iterative way, performing a Fourier transform operation hundreds or thousands of times and then layering on additional types of computation.

We're not just trying to come up with cool methods that beat the state of the art in this controlled lab environment. We actually want to use it in the hospital, with the goal of improving patient outcomes.
Jon Tamir

Those calculations are performed in the Amazon Web Services cloud. The ability to do so as quickly as possible is key not only from a research perspective but also a clinical one. That's because even if the method of taking the raw measurements speeds up the MRI, the clinician still must check the quality of the image while the patient is present.

“If we have a fast scan, but now the reconstruction takes 10 minutes or an hour, then that's not going to be clinically feasible," he said. "We're extending this computation, but we need to do it in a way that maintains efficiency."

In addition to AWS cloud services, Tamir has used AWS Lambda to break the image reconstruction down pixel-by-pixel, sending small bits of data to different Lambda nodes, running the computation, and then aggregating the results.

Related content
Science-based recommendations from the Digital Wellness Lab could inform the development of digital products that help children.

Tamir was already familiar with AWS from his work as a graduate student at UC Berkeley, where he earned his doctorate in electrical engineering. There, he worked with Michael (Miki) Lustig, a professor of electrical engineering and computer science, on using deep learning to reduce knee scan times for patients at Stanford Children's Hospital.

As an undergrad, Tamir explored his interest in digital signal processing through unmanned aerial vehicles (UAVs), working on methods for detecting objects on the ground. After taking Lustig's Principles of MRI course at UC Berkeley, he fell in love with MRI: "It had all of the same mathematical excitement that imaging for UAVs had, but it was also something you could visually see, which was just so cool, and it had a really important societal impact."

Tamir also works with clinicians to understand MRI issues in practice. He and Léorah Freeman, a neurologist who works with multiple sclerosis (MS) patients at UT Health Austin, are trying to figure out how machine learning approaches could make brain scans faster while also detecting attributes that humans might not see.

Related content
Using social media data, the University of Maryland's Philip Resnik aims to help clinicians prioritize individuals who may need immediate attention.

"Tissues that look healthy to the naked eye on the brain MRI may not be healthy if we were to look at them under the microscope," Freeman said. "When we use artificial intelligence, we can look broadly into the brain and try to identify changes that may not be perceptible to the naked eye that can relate to how a patient is doing, how they're going to do in the future, and how they respond to a therapy."

Tamir and Freeman are starting by scanning the brains of healthy volunteers to establish control images to compare with those of MS patients. He hopes that the machine learning method presented at NeurIPS can be tailored to patients with MS at the Dell Medical School in Austin. It could be five to 10 years, he said, before a given method makes its way into standard MRI protocols. But that is Tamir's main goal: clinical adoption.

"We're not just trying to come up with cool methods that beat the state of the art in this controlled lab environment," he said. "We actually want to use it in the hospital, with the goal of improving patient outcomes.”

Research areas

Related content

US, WA, Seattle
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies to deliver game changing value to our customers. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon’s Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon’s on-line retail business. As an economist on our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies with the potential to deliver game changing value to our customers. This is an opportunity for a high-energy individual to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals.
US, CA, San Francisco
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of statistical inference, experimentation design, economic theory and machine learning to design new methods and pricing strategies for assessing pricing innovations. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon's Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon's on-line retail business. As an economist on our team, you will will have the opportunity to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in experimentation design, applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of interns from previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US
The Amazon Supply Chain Optimization Technology (SCOT) organization is looking for an Intern in Economics to work on exciting and challenging problems related to Amazon's worldwide inventory planning. SCOT provides unique opportunities to both create and see the direct impact of your work on billions of dollars’ worth of inventory, in one of the world’s most advanced supply chains, and at massive scale. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Knowledge of econometrics and Stata/R/or Python is necessary, and experience with SQL, Hadoop, and Spark would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Seattle
The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. We are looking for an Intern Economist with excellent coding skills to design and develop rigorous models to assess the causal impact of fees on third party sellers’ behavior and business performance. As a Science Intern, you will have access to large datasets with billions of transactions and will translate ambiguous fee related business problems into rigorous scientific models. You will work on real world problems which will help to inform strategic direction and have the opportunity to make an impact for both Amazon and our Selling Partners.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, Hadoop, Spark and Python would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking interns and co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it! Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, artificial intelligence, human-robot interaction, optimization and more.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking interns and co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it! Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, artificial intelligence, human-robot interaction, optimization and more.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, MA, Boston
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics, a wholly owned subsidiary of Amazon.com, empowers a smarter, faster, more consistent customer experience through automation. Amazon Robotics automates fulfillment center operations using various methods of robotic technology including autonomous mobile robots, sophisticated control software, language perception, power management, computer vision, depth sensing, machine learning, object recognition, and semantic understanding of commands. Amazon Robotics has a dedicated focus on research and development to continuously explore new opportunities to extend its product lines into new areas. AR is seeking uniquely talented and motivated data scientists to join our Global Services and Support (GSS) Tools Team. GSS Tools focuses on improving the supportability of the Amazon Robotics solutions through automation, with the explicit goal of simplifying issue resolution for our global network of Fulfillment Centers. The candidate will work closely with software engineers, Fulfillment Center operation teams, system engineers, and product managers in the development, qualification, documentation, and deployment of new - as well as enhancements to existing - operational models, metrics, and data driven dashboards. As such, this individual must possess the technical aptitude to pick-up new BI tools and programming languages to interface with different data access layers for metric computation, data mining, and data modeling. This role is a 6 month co-op to join AR full time (40 hours/week) from July – December 2023. The Co-op will be responsible for: Diving deep into operational data and metrics to identify and communicate trends used to drive development of new tools for supportability Translating operational metrics into functional requirements for BI-tools, models, and reporting Collaborating with cross functional teams to automate AR problem detection and diagnostics