Multi-objective relevance ranking

By Michinari Momma, Ali Bagheri Garakani, Yi Sun
2019
Download Copy BibTeX
Copy BibTeX
In this paper, we introduce an Augmented Lagrangian based method in a search relevance ranking algorithm to incorporate the multi-dimensional nature of relevance and business constraints, both of which are the requirements for building relevance ranking models in production. The off-the-shelf solutions cannot handle such complex objectives and therefore, modelers are left hand-tuning of parameters that have only indirect impact to the objectives, at-tempting to incorporate multiple objectives (MO) in a model. This process is time-consuming and tends to face suboptimality. The proposed method is designed to systematically solve the MO problem in a constrained optimization framework, which is integrated with a popular Boosting algorithm and is, by all means, a novel contribution. Furthermore, we propose a procedure to specify the constraints to achieve business goals and the exploration scales linearly in the number of constraints, while existing methodology can scale exponentially. The experimental results show that the method successfully builds models that achieve MO criteria much more efficiently than existing methods. The potential impact includes significant reduction in model development time and allows for automation of model refresh even with presence of several MO criteria, in real world production system scale with hundreds of millions of records.

Latest news

US, WA, Bellevue
Are you inspired by invention? Do you want to see how your work impacts the bigger picture? If yes, you'll fit right in at Amazon's Last Mile Simulations and Analytics Engineering team. The WW AMZL Simulations and Analytics Engineering team drives innovation across our Last Mile network delivery stations through digital twin technology and the seamless integration of process engineering, physics, design, technology, and robotics. We start with the customer and work backwards in everything we do. If you're interested in joining a rapidly growing team building a unique, solutions-advisory group with a relentless focus on the customer, you've come to the right place. This is a blue-sky role where you'll roll up your sleeves and dive into big data sets to build digital twins—discrete event simulations, robotics models, and physics-based 3D simulations using tools like Flexsim, AnyLogic, and Emulate 3D. You'll develop experimentation systems at scale, integrating process workflows, physical dynamics, facility design, emerging technologies, and robotic automation to create comprehensive virtual replicas of our operations. Build optimization algorithms and leverage advanced technologies across Amazon. This is an opportunity to think big about solving challenging problems for our customers. As a Sr. Simulation Scientist, you'll deep dive into complex problems and drive relentlessly towards innovative solutions with cross-functional teams. You'll interface and influence various functional teams and individuals at all levels of the organization. Lead strategic digital twin and simulation projects that integrate process design, physics modeling, technology innovation, and robotics to drive operational decisions. Your expertise in synthesizing and communicating insights to audiences of varying technical sophistication will enable you to answer specific business questions and innovate for the future. You'll apply advanced multi-disciplinary methodologies—combining process optimization, physics simulation, design thinking, and robotic systems—to complex use cases across the Last Mile network. You'll also contribute to the end-state vision for digital twin simulation and experimentation of future delivery stations at Amazon. Key job responsibilities • Lead the design, implementation, and delivery of the simulation data science solutions to perform system of systems discrete event simulations for significantly complex operational processes that have a long-term impact on a product, business, or function using FlexSim, Demo 3D, AnyLogic or any other Discrete Event Simulation (DES) software packages • Lead strategic modeling and simulation research projects to drive process design decisions • Be an exemplary practitioner in simulation science discipline to establish best practices and simplify problems to develop discrete event simulations faster with higher standards • Identify and tackle intrinsically hard process flow simulation problems (e.g., highly complex, ambiguous, undefined, with less existing structure, or having significant business risk or potential for significant impact • Deliver artifacts that set the standard in the organization for excellence, from process flow control algorithm design to validation to implementations to technical documents using simulations • Be a pragmatic problem solver by applying judgment and simulation experience to balance cross-organization trade-offs between competing interests and effectively influence, negotiate, and communicate with internal and external business partners, contractors and vendors for multiple simulation projects • Provide simulation data and measurements that influence the business strategy of an organization. Write effective white papers and artifacts while documenting your approach, simulation outcomes, recommendations, and arguments • Lead and actively participate in reviews of simulation research science solutions. You bring clarity to complexity, probe assumptions, illuminate pitfalls, and foster shared understanding within simulation data science discipline • Pay a significant role in the career development of others, actively mentoring and educating the larger simulation data science community on trends, technologies, and best practices • Use advanced statistical /simulation tools and develop codes (python or another object oriented language) for data analysis , simulation, and developing modeling algorithms • Lead and coordinate simulation efforts between internal teams and outside vendors to develop optimal solutions for the network, including equipment specification, material flow control logic, process design, and site layout • Deliver results according to project schedules and quality A day in the life If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply!
US, CA, Sunnyvale
Fauna Robotics is seeking a Principal Applied Scientist to lead the development of evaluation frameworks and data collection protocols that measure, stress-test, and improve robot behavior across real-world tasks. Operating at the intersection of robotics, machine learning, and human-in-the-loop systems, this role owns the infrastructure and methodologies that connect teleoperation, evaluation, and learning — shaping how policies are validated and how high-quality datasets are generated to accelerate system performance. Key job responsibilities What you'll do: - Build and own evaluation frameworks that measure and stress-test robot behavior across real-world tasks - Design data collection protocols that generate high-quality datasets to accelerate system performance - Connect teleoperation, evaluation, and learning systems into cohesive, measurable infrastructure - Translate ambiguous capability goals into actionable, measurable evaluation systems What we're looking for: - A highly experimental, systems-oriented scientist comfortable working across software, robotics, and data pipelines - Deep expertise in machine learning and robotics, with a track record of building evaluation methodologies About the team Fauna Robotics, an Amazon company, is building capable, safe, and genuinely delightful robots for everyday life. Our goal is simple: make robots people actually want to live and interact with in everyday human spaces. We believe that future won’t arrive until building for robotics becomes far more accessible. Today, too much effort is spent reinventing the fundamentals. We’re changing that by developing tightly integrated hardware and software systems that make it faster, safer, and more intuitive to create real-world robotic products. Our work spans the full stack: mechanical design, control systems, dynamic modeling, and intelligent software. The focus is not just functionality, but experience. We’re building robots that feel responsive, expressive, and genuinely useful. At Fauna, you’ll work at the frontier of this space, helping define how robots move, manipulate, and interact with people in natural environments. It’s an opportunity to solve hard problems across hardware and software with a team focused on making robotics accessible and joyful to build. If you care about making robotics real for everyone and building systems that are as delightful as they are capable, we’re interested in hearing from you. an opportunity to solve hard problems across hardware and software with a team focused on making robotics accessible and joyful to build. If you care about making robotics real for everyone and building systems that are as delightful as they are capable, we’re interested in hearing from you.
US, WA, Seattle
Amazon's eCommerce Foundation (eCF) organization provides the core technologies that drive and power Amazon's Stores, Digital, and Other (SDO) businesses. Millions of customer page views and orders per day are enabled by the systems eCF builds from the ground up. CloudTune, within eCF, empowers growth and business agility needs by automatically and efficiently managing AWS capacity and business processes needed to safely meet Amazon's customer demand. CloudTune serves its primary customers, internal software teams, through forecast-driven automation of cost controllership, capacity management, and scaling. We predict expected load, and drive procurement and allocation of AWS capacity for new product launches and high-velocity events like Prime Day and Cyber Monday. CloudTune is looking for a Data Scientist to join our forecasting and optimization team. The team develops sophisticated algorithms that combine machine learning-based demand forecasting with mathematical optimization to solve large-scale capacity planning problems under uncertainty. We work with massive datasets — spanning thousands of availability zone and instance family combinations across global regions — to determine optimal resource allocation strategies that balance infrastructure holding costs against service availability requirements. These models directly inform multi-million dollar capacity investment decisions and drive automated procurement, retention, and release policies across Amazon's compute infrastructure. As a Data Scientist in CloudTune, you will work with other scientists, software engineers, data engineers, and product managers on a variety of important research problems in the areas of stochastic optimization, time series modeling, and operations research. You will work on statistical problems with a high level of ambiguity. You will analyze and process large amounts of data, develop new algorithms and improve existing approaches based on statistical models, machine learning algorithms and big data solutions to automatically scale Amazon’s compute infrastructure, optimizing the balance between availability risk and cost efficiency for all of Amazon businesses. Key job responsibilities - Process and analyze large data sets, mining additional data sources as needed - Build statistical models and drive scalable solutions for multi-year capacity demand forecasting horizons - Create, enhance, and maintain technical documentation, and present to other scientists and business leaders - Process and analyze large-scale operational data to identify patterns and inform model development - Create, enhance, and maintain technical documentation, and present research findings to scientists, engineering teams, and senior leadership
US, WA, Seattle
Pricing is one of the most consequential decisions Amazon makes — and the science behind it needs to be causally rigorous, not just predictive. The P2 Optimization Science (P2OS) team builds the machine learning systems that power Amazon's pricing decisions at scale: demand lift models, customer lifetime value frameworks, and the experimentation infrastructure that validates whether our pricing changes actually work. We're hiring an Applied Scientist to own causal inference at the intersection of ML and pricing experimentation. This role exists because our team has identified a real gap: the methodological bridge between econometric analysis (owned by our economists) and production-scale ML pipelines (owned by our engineers) needs a practitioner who lives in both worlds. You'll build CATE estimation models, design analysis workflows for pricing weblabs, and develop the reusable causal ML infrastructure that the broader team — including non-ML scientists — can rely on. This is not a research role. The bias here is toward shipping production-quality causal pipelines with real downstream business impact. You'll measure success by what changes in LTV estimates, what pricing errors your models help avoid, and whether the economists on your team can actually use what you build. If you're a scientist who wants to work on hard causal identification problems in a high-stakes production environment — and who finds satisfaction in making rigorous methods accessible to a broader team — this role is for you.
US, MA, North Reading
Amazon Robotics is transforming warehouse automation through edge AI and machine learning applied to real-world robotics challenges. We're seeking a Research Scientist to advance our mobile manipulation capabilities by developing novel learning-based approaches that enable robots to navigate and manipulate objects in dynamic fulfillment environments. This role offers the opportunity to conduct original research and translate state-of-the-art findings into production systems operating at Amazon's unprecedented scale. Key job responsibilities Research and Algorithm Development: Formulate novel research problems in robot learning and manipulation, design new model architectures, validate hypotheses through rigorous experimentation, and advance the state of the art in learning-based robotics. Data Strategy and Pipeline Design: Define data requirements for research initiatives, design scalable collection and curation strategies, establish governance and provenance standards, and build reusable pipelines ensuring data quality and reproducibility. Experimentation and Scientific Validation: Design and execute experiments in simulation and real-world embodiments, develop evaluation methodologies and benchmarks, perform ablation studies and statistical analyses, and iterate systematically to advance model performance. Prototyping and Research Infrastructure: Develop clean, well-documented research codebases, build experimentation frameworks and evaluation tooling, contribute to shared training infrastructure, and implement interfaces for broader robotics integration. Scientific Leadership and Publication: Drive an independent research agenda aligned with team objectives, publish at top-tier venues (e.g., RSS, CoRL, ICRA, NeurIPS), identify research gaps through literature reviews, and present findings via technical reports and talks. Cross-Functional Collaboration: Partner with scientists, engineers, and leaders across teams to translate research into deployable solutions, mentor junior researchers, contribute to the team's scientific culture, and support integration with robotics hardware teams. A day in the life If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! About the team Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart, collaborative team of enthusiastic doers that work passionately to apply innovative advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even image yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun!
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Amazon's advertising portfolio helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! The Creative X team within Amazon Advertising time aims to democratize access to high-quality creatives (text, audio, images, videos) by building AI-driven solutions for advertisers. To accomplish this, we are investing in latent-diffusion models, large language models (LLM), generative audio (music and speech synthesis), computer vision (CV), reinforced learning (RL) and related methods. You will be part of a close-knit team of applied scientists and product managers who are highly collaborative and at the top of their respective fields. We are looking for talented Applied Scientists who have a proven record of developing Generative AI science at the highest standards of innovation, and experience working and delivering product-ready code, are adept at a variety of skills, especially with LLMs, generative audio, computer vision, latent diffusion or related foundational models that will accelerate our plans to generate high-quality creatives on behalf of advertisers. Every member of the team is expected to build customer (advertiser) facing features, contribute to the collaborative spirit within the team, publish, patent, and bring innovation research to raise the bar within the team.
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing businesses. Amazon's advertising portfolio helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! The Creative X team within Amazon Advertising time aims to democratize access to high-quality creatives (audio, images, videos, text) by building AI-driven solutions for advertisers. To accomplish this, we are investing in understanding how best users can leverage Generative AI methods such as latent-diffusion models, large language models (LLM), generative audio (music and speech synthesis), computer vision (CV), reinforced learning (RL) and related. As an Applied Scientist you will be part of a close-knit team of other applied scientists, engineers and product managers who are highly collaborative and at the top of their respective fields. We are looking for talented Applied Scientists who are adept at a variety of skills, especially at the interface of how users can use state-of-the-art generative music and audio, computer vision, latent diffusion or related foundational models that will accelerate our plans to generate high-quality creatives on behalf of advertisers. Every member of the team is expected to build customer (advertiser) facing features, contribute to the collaborative spirit within the team, publish, patent, and bring innovative research to raise the bar within the team. You will: - Drive the invention and development of novel methods for the use of Generative AI in ads - Work closely and integrate end-to-end proof-of-concept Machine Learning projects that have a high degree of ambiguity, scale and complexity. - Build interface-oriented systems that use Machine Learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. - Run A/B experiments, gather data, and perform statistical analysis. - Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. - Mentor and help recruit Applied Scientists to the team. - Present results and explain methods to senior leadership. - Willingness to publish research at internal and external top scientific venues. - Write and pursue IP submissions. Key job responsibilities This role is focused on developing Machine Learning and Generative AI methods to augment generative imagery and videos. You will develop new paradigms that will impact how users can engage and benefit from GenAI and that are at the core of advertising-facing tools that we are launching. You will also work on development of ML and GenAI models. You will conduct literature reviews to stay on the top of the field. You will regularly engage with product managers, who will partner with you to productize your work. About the team
US, NY, New York
We are seeking an Applied Scientist to develop and optimize Visual Inertial Odometry (VIO) and sensor fusion systems for our intelligent robots. In this role, you will design, implement, and deploy state estimation and tracking algorithms that enable robots to understand their position and motion in real time, even in challenging and dynamic environments. You will own the full pipeline from algorithm development through embedded deployment, ensuring that perception systems run efficiently on resource-constrained robotic hardware. You will also leverage modern machine learning approaches to push the boundaries of classical perception methods, combining learned representations with geometric techniques to achieve robust, real-time performance. This is a deeply hands-on role. You will work directly with sensors, hardware, and real-world data, while prototyping, testing, and iterating in physical environments. The ideal candidate has strong foundations in VIO and sensor fusion, practical experience optimizing algorithms for embedded platforms, and familiarity with how modern deep learning is transforming perception. Key job responsibilities - Design and implement Visual Inertial Odometry algorithms for robust real-time state estimation on robotic platforms like Sprout - Develop multi-sensor fusion pipelines integrating cameras, IMUs, and other sensing modalities for accurate pose tracking - Optimize perception and tracking algorithms for deployment on embedded hardware (e.g., ARM, GPU-accelerated edge devices) under strict latency and power constraints - Apply modern ML-based perception techniques (learned features, depth estimation, neural odometry) to complement and improve classical geometric approaches - Build and maintain calibration, evaluation, and benchmarking infrastructure for perception systems - Collaborate with hardware, controls, and navigation teams to integrate perception outputs into the robot’s autonomy stack - Lead technical projects from research prototyping through production deployment
US, NY, New York
The FAIM (Full Funnel Agentic Intelligence and Models) builds the agentic AI systems behind next-generation advertising. The team develops multimodal foundation models and LLM-driven agents that power shopper understanding, audience-aware creative generation, and agentic advertising workflows. As a Principal Applied Scientist for the FAIM team, you will invent the models that help advertisers achieve desired outcomes. Today's state of the art models can setup ad campaigns to a specification, but do not understand advertising strategy well enough to help with abstract goals that real advertisers ask about like "how can I make more money." This is a rare charter to build foundational optimization science where little exists today, spanning campaign recommendation, cross-product budget allocation, incrementality measurement, and long-term-sales modeling. You will set technical direction for a growing team, partner with engineering and product to take models from research to production at Amazon scale, and directly move advertiser ROAS and new-to-brand growth. In this role you will - Own the science vision for training models in our RL Gym from end-to-end - Invent models for joint budget allocation, incrementality, and long-term value - Take innovations from prototype to production serving live advertiser campaigns - Raise the technical bar across scientists and engineers Key job responsibilities - Define the long-term scientific vision for our RL Gym based training process focusing on campaign optimization and other advertising workflows - Translating ambiguous advertiser needs and competing objectives into a concrete science roadmap. - Invent, prototype, and productionize machine learning and optimization solutions for joint budget allocation across sponsored ad products, spanning the shopper journey from awareness to purchase. - Develop rigorous approaches to incrementality measurement and long-term-sales modeling that ground optimization in true advertiser value. - Design and lead large-scale experiments and analyses to validate hypotheses and guide product direction. - Partner closely with engineering and product to define technical contracts, data schemas, and serving systems that carry models into production. - Raise the technical bar across science and engineering through mentorship, design reviews, and hands-on collaboration. - Grow scientific talent and publish impactful research internally and at top-tier venues.
US, WA, Seattle
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The SP team's primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! Within Sponsored Products, the Bidding team is responsible for defining and delivering a collection of advertising products around bid controls (dynamic bidding, bid recommendations, etc.) that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven fundamentally from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. Key job responsibilities • Lead a new initiative across Sponsored Products Bidding focused on AI/ML based features. • Be the technical leader in AI, Machine Learning; lead efforts within this team and across other teams. • Perform hands-on analysis and modeling of enormous data sets to develop insights that increase traffic monetization and merchandise sales, without compromising the shopper experience. • Drive end-to-end AI/Machine Learning projects that have a high degree of ambiguity, scale, complexity. • Build models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your AI/ML models. • Run A/B experiments, gather data, and perform statistical analysis. • Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. • Research new and innovative AI/ machine learning approaches. A day in the life Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon's Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. About the team The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through the latest generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. The SPB Bidding team within Sponsored Products and Brands is focused on guiding and supporting Millions of advertisers to meet their advertising needs of creating and managing ad campaigns. At this scale, the complexity of diverse advertiser goals, campaign types, and market dynamics creates both a massive technical challenge and a transformative opportunity: even small improvements in bidding systems can have outsized impact on advertiser success and Amazon’s retail ecosystem. Our vision is to build a highly personalized, context-aware bidding system that leverages auction simulations, ML models, and optimization algorithms. This framework, will operate across SPB bidding system and proactively delivering value based on deep understanding of the advertiser. To execute this vision, we collaborate closely with stakeholders across Ad Console, Sales, and Marketing to identify opportunities—from high-level product guidance down to granular recommendations—and deliver them through a tailored, personalized experience. Our work is grounded in state-of-the-art bidding agent architectures, tool integration, reasoning frameworks, and model customization approaches (including tuning and preference optimization), ensuring our systems are both scalable and adaptive.