In any given month as a principal data scientist at Amazon Web Services (AWS), Elena Ehrlich might be working on time-series modeling, a computer vision project, a natural language processing problem, and more. Her work within the AWS Professional Services organization involves solving data problems for customers in fields ranging from media to energy to sports.
Sometimes a customer has a particular model in mind and will consult with Ehrlich's team to build or refine it. Often, though, they are not so far along. They simply have a business problem to solve. Ehrlich works with them anywhere from two months to three years to develop a solution that the client can then maintain going forward.
Ehrlich likes the ability to apply data science across a variety of verticals without having to switch jobs, or even teams.
"Amazon has a large and diverse landscape of customers, so I can gain a lot of different domain knowledge," Ehrlich says. "Each customer's needs are unique, which makes it very interesting, and the challenge is to come up with solutions that can be reused by other customers as well.”
Better predictions for spiky time series
Ehrlich's work with the NFL is one example of science applied to business challenges. Independent of her team's existing work with the league, she and colleagues developed a method for modeling heavy-tailed time series. In these sequences, one can have dramatic, unpredictable spikes: Think extreme rainfall events that shape totals over the course of a year, or a product suddenly going viral, increasing demand.
Many statistical methods that would perform fine on more uniform curves falter when it comes to the noise of heavy-tailed time series. Yet being able to characterize these tails is important. On an EKG, for instance, you must be able to tell whether a peak in heart rate signals disease or simply the beginning of a workout.
Predictive models were not able to reliably pinpoint such anomalies. Over the course of a few months, Ehrlich and Amazon researchers Francois-Xavier Aubet and Laurent Callot developed a solution, which they presented at the 2021 International Conference on Learning Representations' RobustML Workshop.
"If you're seeing an issue from a few customers, then it's worth zooming out, solving it as a research problem, and then going back to examine to which customers this can apply," Ehrlich said.
Their solution, the Spliced Binned-Pareto method, combines two statistical techniques, the binned and the Pareto distributions. The latter stems from Italian economist Vilfredo Pareto's 80/20 rule: The 1920s idea that 80% of outcomes emerge from 20% of the causes (most of a country's wealth attributable to a fifth of its population, for example). This power-law relationship, when generalized, delivered the Second Theorem of Extreme Value Theory in 1975, which states that any and all distribution tails can be well approximated by a Generalized Pareto Distribution.
The researchers combined this with binned distribution, which discretizes regions within a larger dataset. Their method effectively cordons off and zeroes in on the spikes within a time series, leading to an improved ability to accommodate these extremes and calibrate estimates of them over time, which in turn leads to more accurate heavy-tail predictions.
This work aligned with a request from the NFL. While quarterback ratings exist in various forms, the league wanted a metric to rate passing performance. But a meaningful passing metric had to extend beyond passing yards, touchdowns, and interceptions to reflect the degree of difficulty for those outcomes given the specific play’s circumstances, in order to evaluate the NFL quarterback’s performance.
The resulting NFL Passing Score, developed using Ehrlich's Spliced Binned-Pareto method, can place a quarterback's performance within the context of expected performance across the league.
That's because it is capable of estimating that heavy tail — in this case, those exceptional moments in a quarterback's throw — and assign them the proper weight toward the overall score. The NFL debuted the Passing Score early this year, ahead of the Super Bowl. Perhaps not surprisingly, Green Bay Packers quarterback Aaron Rodgers had the highest score.
An early affinity for math
Though the Passing Score project is complete, Ehrlich continues to refine the Spliced Binned-Pareto distribution, among other facets of her work.
"You want to forge a team to be at the leading edge of industry," she says. "Leading edge is determined by how short the lag is between your academic progress and marketplace usage."
Ehrlich has bachelor's and master's degrees in mathematics and a doctorate degree in statistics, all from Imperial College London — a school she chose in part because its program allowed her to focus solely on mathematics from her first year of university.
The predilection toward math and science runs in the family: Ehrlich's father is a mathematician, her mother a physicist. Numbers were a part of childhood, growing up in New Jersey. On skiing trips, she recalls, her father would point out the numbers on chair lifts and challenge her to factor them or shout "prime." If she got them all right, she'd get a candy reward.
"I thought these were fun games," Ehrlich says.
Ehrlich is glad to have had a singular focus on mathematics from undergrad onward.
"My career is where I went to get breadth and width, but it was really helpful to have this depth," she says. "It helps me learn faster when I get to a new domain or application, just having the technical strength."
'Genuinely excited about the problems'
Ehrlich embarked on her PhD, which focused on state-space models with applications for aerospace and missile tracking, thinking she would stay in academia. But as she completed the work in 2014, she knew there was an emerging job market for skills like hers.
"The real world had an appreciation for methodologies that weren't exactly instant," she says. "It takes some time to research good solutions and test out their longevity and experiment their generalizability."
She held research scientist positions at companies including IBM and Winton Capital Management before joining Amazon in 2017. Heading to Seattle headquarters, she prepared for some housekeeping-type meetings. She was surprised to find opportunities to sign up for dev ops and other classes related to Amazon technology. It felt like being back at university, she says, in a good way. The culture reflects this drive to learn; in fact, one of the company’s leadership principles is “learn and be curious.”
"People that come to work here are genuinely excited about the problems," she says. "It makes for more data-driven conversation based on the problem at hand. That intersects with the fact that since Amazon is big with a wide-range of opportunities, it attracts a lot of top talent."
An early project for Ehrlich focused on 21st Century Fox. She developed and implemented an optimized Ad Sales Pricing platform for the company's advertising time spots, horizontally scaling to match potential advertisers with spots across millions of opportunities. Working with sales and engineering teams, she moved the algorithm into production, boosting revenue for Fox.
For students who are interested in a career like Ehrlich's, she recommends starting with first principles and then confirming that understanding with actual projects.
"You should have some corner of theory really well understood. Then iterate between knowing something and implementing it — it doesn't even matter how small," she says. "This is a very fast, but very solid, way to grow."
