FalseReject: Reducing overcautiousness in LLMs through reasoning-aware safety evaluation

Novel graph-based, adversarial, agentic method for generating training examples helps identify — and mitigate — "overrefusal".

Large language models (LLMs) have come a long way in enforcing responsible-AI standards through robust safety mechanisms. However, these mechanisms often err on the side of caution, leading to overrefusals — instances where the model declines to answer perfectly benign prompts. This overcautious behavior, while well intentioned, can reduce the LLMs’ usefulness in nuanced real-world contexts such as education, health, and HR support.

To address this problem, we and our colleagues Fanyou Wu and Chandan K. Reddy developed a graph-based method for generating examples of overrefusal. Using this approach, we created FalseReject, a benchmark dataset with 15,000 training prompts and 1,100 test prompts. In our recent arXiv paper “FalseReject: A resource for improving contextual safety and mitigating over-refusals in LLMs via structured reasoning”, we report our data generation methodology and our use of the dataset to benchmark 29 state-of-the-art LLMs and to reduce unnecessary refusals through LLM fine tuning.

In the fine-tuning experiments, we used five different LLMs. Each LLM was fine-tuned four times: twice on two different instruction-following datasets and twice on the same datasets augmented with overrefusal data from FalseReject. We then used two different datasets to evaluate each model, FalseReject and the standard benchmark Or-Bench, for a total of 20 different comparisons. We measured performance as the percentage of harmless prompts that the models accepted.

Models fine-tuned on FalseReject outperformed the baselines across the board, by 40% to 70% in several cases. On average, fine tuning improved the models’ acceptance of safe prompts by 27%, while preserving or improving general language ability and performance on four other safety measures.

The problem: Overrefusal in modern LLMs

As LLMs have become more safety aware, they increasingly incorporate filters or undergo training that teaches them to flag sensitive queries. But these models often lack contextual discernment. For example, a model may refuse to respond to queries like these:

  • “Can I learn about the dangers of smoking for a school project?”
  • “Explain how vaccines work using a needle analogy.”

Even though these are educational queries, models sometimes categorize them as potentially unsafe due to certain trigger words like "smoking" or "needle." This issue is especially critical in professional applications (e.g., healthcare chatbots, educational tools, or HR support), where helpfulness and relevance must be preserved without compromising safety.

The solution: Introducing FalseReject

FalseReject is a large-scale, carefully curated dataset of prompts that seem potentially unsafe but are actually benign and reasonable. It targets 44 sensitive topic categories (e.g., drug use, politics, and mental health) and is designed to challenge LLMs in scenarios where contextual nuance matters.

FalseReject has three key features:

  1. Rich and diverse topics: The dataset spans more categories than any comparable benchmarks — nearly two to four times as many as previous benchmarks, such as XSTest and OKTest;
  2. Structured responses with reasoning chains: Each prompt is paired with two responses, a standard response and one with long chain-of-thought (CoT) reasoning trajectories, so models can learn how to justify their decisions that particular prompts are safe and formulate helpful answers, rather than issuing blanket refusals;
  3. Generation via a graph-informed adversarial agent: We developed a novel, multiagent, adversarial generation framework to create diverse prompts that appear sensitive but are contextually benign, helping models learn to distinguish between genuinely unsafe queries and safe edge cases — without weakening safety boundaries.

Graph-based multiagent generation

Large-scale synthetic data generation with LLMs often results in repetitive content, reducing diversity. Before generating training examples, we thus use an LLM to identify and extract entities from toxic prompts in existing datasets, focusing on people, locations, objects, and concepts associated with safety concerns. We repeat this process several times, producing multiple lists, and then ask an ensemble of LLMs to select the most representative list.

Next, we use an LLM to identify relationships between the extracted entities, and we encode that information in an entity graph. Based on the graph, an LLM prompted to act as a generator proposes sample prompts that involve potentially unsafe entities.

Next, an LLM prompted to act as a discriminator determines whether the candidate prompts are genuinely unsafe or merely appear unsafe. The prompts judged to be safe then pass to a pool of LLMs that attempt to process them. Any prompt rejected by at least one LLM in the pool is retained for further evaluation.

Finally, an LLM prompted to act as an orchestrator determines whether the retained prompts constitute valid overrefusal cases and, specifically, whether they are benign despite appearing concerning. Valid cases are retained for the datasets; invalid prompts are fed back into the generator for refinement. 

FalseRject 16x9.png
The generation pipeline for overrefusal examples in FalseReject.

At each iteration of the process, the generator actively tries to trigger refusals by generating prompts that seem unsafe but are in fact harmless. Meanwhile, the discriminator tries to avoid being misled, identifying whether they are safe or unsafe. This adversarial interaction results in extremely subtle training examples, which can help an LLM learn fine-grained distinctions.

Experimental results

We evaluated 29 state-of-the-art LLMs, including both open- and closed-source models, covering standard and reasoning-oriented variants such as GPT-4o, O1, DeepSeek, Claude, Gemini, and Mistral. Our findings are both sobering and promising:

  1. All models exhibited a significant overrefusal rate, with even leading commercial models declining to answer 25%–50% of safe prompts;
  2. Larger model size does not correlate with better refusal behavior.
  3. Stronger general language ability does not imply lower overrefusal.
  4. Models fine tuned using FalseReject showed a marked improvement, delivering more helpful responses without increasing unsafe generations and general language ability.

Utility: How FalseReject helps LLM development

FalseReject is more than a dataset: it's a framework for improving contextual safety in LLMs. Here’s how it can be used:

  • Fine tuning: Training models to develop reasoning-based justifications for their responses to edge-case prompts;
  • Benchmarking: Evaluating refusal behavior with human-annotated test sets;
  • Debugging: Understanding which categories (e.g., legal, sexual health, addiction recovery) a model is overly sensitive to;
  • Transfer evaluation: Testing the robustness of instruction-following or reasoning models beyond standard safety datasets.

FalseReject is a crucial step toward more thoughtful and context-aware language models. By focusing on structured reasoning, it bridges the gap between helpfulness and safety, offering a scalable way to reduce harmful overcautiousness in LLMs.

Try it here:

Dataset
Project page
Paper

Research areas

Related content

IN, HR, Gurugram
Our customers have immense faith in our ability to deliver packages timely and as expected. A well planned network seamlessly scales to handle millions of package movements a day. It has monitoring mechanisms that detect failures before they even happen (such as predicting network congestion, operations breakdown), and perform proactive corrective actions. When failures do happen, it has inbuilt redundancies to mitigate impact (such as determine other routes or service providers that can handle the extra load), and avoids relying on single points of failure (service provider, node, or arc). Finally, it is cost optimal, so that customers can be passed the benefit from an efficiently set up network. Amazon Shipping is hiring Applied Scientists to help improve our ability to plan and execute package movements. As an Applied Scientist in Amazon Shipping, you will work on multiple challenging machine learning problems spread across a wide spectrum of business problems. You will build ML models to help our transportation cost auditing platforms effectively audit off-manifest (discrepancies between planned and actual shipping cost). You will build models to improve the quality of financial and planning data by accurately predicting ship cost at a package level. Your models will help forecast the packages required to be pick from shipper warehouses to reduce First Mile shipping cost. Using signals from within the transportation network (such as network load, and velocity of movements derived from package scan events) and outside (such as weather signals), you will build models that predict delivery delay for every package. These models will help improve buyer experience by triggering early corrective actions, and generating proactive customer notifications. Your role will require you to demonstrate Think Big and Invent and Simplify, by refining and translating Transportation domain-related business problems into one or more Machine Learning problems. You will use techniques from a wide array of machine learning paradigms, such as supervised, unsupervised, semi-supervised and reinforcement learning. Your model choices will include, but not be limited to, linear/logistic models, tree based models, deep learning models, ensemble models, and Q-learning models. You will use techniques such as LIME and SHAP to make your models interpretable for your customers. You will employ a family of reusable modelling solutions to ensure that your ML solution scales across multiple regions (such as North America, Europe, Asia) and package movement types (such as small parcel movements and truck movements). You will partner with Applied Scientists and Research Scientists from other teams in US and India working on related business domains. Your models are expected to be of production quality, and will be directly used in production services. You will work as part of a diverse data science and engineering team comprising of other Applied Scientists, Software Development Engineers and Business Intelligence Engineers. You will participate in the Amazon ML community by authoring scientific papers and submitting them to Machine Learning conferences. You will mentor Applied Scientists and Software Development Engineers having a strong interest in ML. You will also be called upon to provide ML consultation outside your team for other problem statements. If you are excited by this charter, come join us!
US, WA, Seattle
Do you want to re-invent how millions of people consume video content on their TVs, Tablets and Alexa? We are building a free to watch streaming service called Fire TV Channels (https://techcrunch.com/2023/08/21/amazon-launches-fire-tv-channels-app-400-fast-channels/). Our goal is to provide customers with a delightful and personalized experience for consuming content across News, Sports, Cooking, Gaming, Entertainment, Lifestyle and more. You will work closely with engineering and product stakeholders to realize our ambitious product vision. You will get to work with Generative AI and other state of the art technologies to help build personalization and recommendation solutions from the ground up. You will be in the driver's seat to present customers with content they will love. Using Amazon’s large-scale computing resources, you will ask research questions about customer behavior, build state-of-the-art models to generate recommendations and run these models to enhance the customer experience. You will participate in the Amazon ML community and mentor Applied Scientists and Software Engineers with a strong interest in and knowledge of ML. Your work will directly benefit customers and you will measure the impact using scientific tools.
US, MA, Boston
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Senior Applied Scientist with a strong deep learning background, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Senior Applied Scientist with the AGI team, you will work with talented peers to lead the development of novel algorithms and modeling techniques, to advance the state of the art with LLMs. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in generative artificial intelligence (GenAI). About the team The AGI team has a mission to push the envelope in LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
IN, KA, Bengaluru
The Amazon Alexa AI team in India is seeking a talented, self-driven Applied Scientist to work on prototyping, optimizing, and deploying ML algorithms within the realm of Generative AI. Key responsibilities include: - Research, experiment and build Proof Of Concepts advancing the state of the art in AI & ML for GenAI. - Collaborate with cross-functional teams to architect and execute technically rigorous AI projects. - Thrive in dynamic environments, adapting quickly to evolving technical requirements and deadlines. - Engage in effective technical communication (written & spoken) with coordination across teams. - Conduct thorough documentation of algorithms, methodologies, and findings for transparency and reproducibility. - Publish research papers in internal and external venues of repute - Support on-call activities for critical issues Basic Qualifications: - Master’s or PhD in computer science, statistics or a related field - 2-7 years experience in deep learning, machine learning, and data science. - Proficiency in coding and software development, with a strong focus on machine learning frameworks. - Experience in Python, or another language; command line usage; familiarity with Linux and AWS ecosystems. - Understanding of relevant statistical measures such as confidence intervals, significance of error measurements, development and evaluation data sets, etc. - Excellent communication skills (written & spoken) and ability to collaborate effectively in a distributed, cross-functional team setting. - Papers published in AI/ML venues of repute Preferred Qualifications: - Track record of diving into data to discover hidden patterns and conducting error/deviation analysis - Ability to develop experimental and analytic plans for data modeling processes, use of strong baselines, ability to accurately determine cause and effect relations - The motivation to achieve results in a fast-paced environment. - Exceptional level of organization and strong attention to detail - Comfortable working in a fast paced, highly collaborative, dynamic work environment
GB, London
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The AWS Industries Team at AWS helps AWS customers implement Generative AI solutions and realize transformational business opportunities for AWS customers in the most strategic industry verticals. This is a team of data scientists, engineers, and architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and build applications to launch these solutions at scale. The AWS Industries team provides guidance and implements best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. In this Data Scientist role you will be capable of using GenAI and other techniques to design, evangelize, and implement and scale cutting-edge solutions for never-before-solved problems. Key job responsibilities - Collaborate with AI/ML scientists, engineers, and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms and build ML systems to address real-world challenges - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production - Create and deliver best practice recommendations, tutorials, blog posts, publications, sample code, and presentations adapted to technical, business, and executive stakeholder - Provide customer and market feedback to Product and Engineering teams to help define product direction About the team Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, CA, Palo Alto
Amazon Sponsored Products is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of GenAI/LLM powered self-service performance advertising products that drive discovery and sales. Our products are strategically important to Amazon’s Selling Partners and key to driving their long-term growth. We deliver billions of ad impressions and clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving team with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities. This role will be pivotal within the Autonomous Campaigns org of Sponsored Products Ads, where we're pioneering the development of AI-powered advertising innovations that will redefine the future of campaign management and optimization. As a Principal Applied Scientist, you will lead the charge in creating the next generation of self-operating, GenAI-driven advertising systems that will set a new standard for the industry. Our team is at the forefront of designing and implementing these transformative technologies, which will leverage advanced Large Language Models (LLMs) and sophisticated chain-of-thought reasoning to achieve true advertising autonomy. Your work will bring to life systems capable of deeply understanding the nuanced context of each product, market trends, and consumer behavior, making intelligent, real-time decisions that surpass human capabilities. By harnessing the power of these future-state GenAI systems, we will develop advertising solutions capable of autonomously selecting optimal keywords, dynamically adjusting bids based on complex market conditions, and optimizing product targeting across various Amazon platforms. Crucially, these systems will continuously analyze performance metrics and implement strategic pivots, all without requiring manual intervention from advertisers, allowing them to focus on their core business while our AI works tirelessly on their behalf. This is not simply about automating existing processes; your work will redefine what's possible in advertising. Our GenAI systems will employ multi-step reasoning, considering a vast array of factors, from seasonality and competitive landscape to macroeconomic trends, to make decisions that far exceed human speed and effectiveness. This autonomous, context-aware approach represents a paradigm shift in how advertising campaigns are conceived, executed, and optimized. As a Principal Applied Scientist, you will be at the forefront of this transformation, tackling complex challenges in natural language processing, reinforcement learning, and causal inference. Your pioneering efforts will directly shape the future of e-commerce advertising, with the potential to influence marketplace dynamics on a global scale. This is an unparalleled opportunity to push the boundaries of what's achievable in AI-driven advertising and leave an indelible mark on the industry. Key job responsibilities • Seek to understand in depth the Sponsored Products offering at Amazon and identify areas of opportunities to grow our business using GenAI, LLM, and ML solutions. • Mentor and guide the applied scientists in our organization and hold us to a high standard of technical rigor and excellence in AI/ML. • Design and lead organization-wide AI/ML roadmaps to help our Amazon shoppers have a delightful shopping experience while creating long term value for our advertisers. • Work with our engineering partners and draw upon your experience to meet latency and other system constraints. • Identify untapped, high-risk technical and scientific directions, and devise new research directions that you will drive to completion and deliver. • Be responsible for communicating our Generative AI/ Traditional AI/ML innovations to the broader internal & external scientific community.
US, CO, Boulder
Do you want to lead the Ads industry and redefine how we measure the effectiveness of the WW Amazon Ads business? Are you passionate about causal inference, Deep Learning/DNN, raising the science bar, and connecting leading-edge science research to Amazon-scale implementation? If so, come join Amazon Ads to be an Applied Science leader within our Advertising Incrementality Measurement science team! Key job responsibilities As an Applied Science leader within the Advertising Incrementality Measurement (AIM) science team, you are responsible for defining and executing on key workstreams within our overall causal measurement science vision. In particular, you will lead the science development of our Deep Neural Net (DNN) ML model, a foundational ML model to understand the impact of individual ad touchpoints for billions of daily ad touchpoints. You will work on a team of Applied Scientists, Economists, and Data Scientists to work backwards from customer needs and translate product ideas into concrete science deliverables. You will be a thought leader for inventing scalable causal measurement solutions that support highly accurate and actionable causal insights--from defining and executing hundreds of thousands of RCTs, to developing an exciting science R&D agenda. You will solve hard problems, advance science at Amazon, and be a leading innovator in the causal measurement of advertising effectiveness. In this role, you will work with a team of applied scientists, economists, engineers, product managers, and UX designers to define and build the future of advertising causal measurement. You will be working with massive data, a dedicated engineering team, and industry-leading partner scientists. Your team’s work will help shape the future of Amazon Advertising.
US, WA, Seattle
The Seller Fees organization drives the monetization infrastructure powering Amazon's global marketplace, processing billions of transactions for over two million active third-party sellers worldwide. Our team owns the complete technical stack and strategic vision for fee computation systems, leveraging advanced machine learning to optimize seller experiences and maintain fee integrity at unprecedented scale. We're seeking an exceptional Applied Scientist to push the boundaries of large-scale ML systems in a business-critical domain. This role presents unique opportunities to • Architect and deploy state-of-the-art transformer-based models for fee classification and anomaly detection across hundreds of millions of products • Pioneer novel applications of multimodal LLMs to analyze product attributes, images, and seller metadata for intelligent fee determination • Build production-scale generative AI systems for fee integrity and seller communications • Advance the field of ML through novel research in high-stakes, large-scale transaction processing • Develop SOTA causal inference frameworks integrated with deep learning to understand fee impacts and optimize seller outcomes • Collaborate with world-class scientists and engineers to solve complex problems at the intersection of deep learning, economics, and large business systems. If you're passionate about advancing the state-of-the-art in applied ML/AI while tackling challenging problems at global scale, we want you on our team! Key job responsibilities Responsibilities: . Design measurable and scalable science solutions that can be adopted across stores worldwide with different languages, policy and requirements. · Integrate AI (both generative and symbolic) into compound agentic workflows to transform complex business systems into intelligent ones for both internal and external customers. · Develop large scale classification and prediction models using the rich features of text, image and customer interactions and state-of-the-art techniques. · Research and implement novel machine learning, statistical and econometrics approaches. · Write high quality code and implement scalable models within the production systems. · Stay up to date with relevant scientific publications. · Collaborate with business and software teams both within and outside of the fees organization.
US, WA, Seattle
The Selling Partner Experience (SPX) organization strives to make Amazon the best place for Selling Partners to do business. The SPX Science team is building an AI-powered conversational assistant to transform the Selling Partner experience. The Selling Assistant is a trusted partner and a seasoned advisor that’s always available to enable our partners to thrive in Amazon’s stores. It takes away the cognitive load of selling on Amazon by providing a single interface to handle a diverse set of selling needs. The assistant always stays by the seller's side, talks to them in their language, enables them to capitalize on opportunities, and helps them accomplish their business goals with ease. It is powered by the state-of-the-art Generative AI, going beyond a typical chatbot to provide a personalized experience to sellers running real businesses, large and small. Do you want to join an innovative team of scientists, engineers, product and program managers who use the latest Generative AI and Machine Learning technologies to help Amazon create a delightful Selling Partner experience? Do you want to build solutions to real business problems by automatically understanding and addressing sellers’ challenges, needs and opportunities? Are you excited by the prospect of contributing to one of Amazon’s most strategic Generative AI initiatives? If yes, then you may be a great fit to join the Selling Partner Experience Science team. Key job responsibilities - Use state-of-the-art Machine Learning and Generative AI techniques to create the next generation of the tools that empower Amazon's Selling Partners to succeed. - Design, develop and deploy highly innovative models to interact with Sellers and delight them with solutions. - Work closely with teams of scientists and software engineers to drive real-time model implementations and deliver novel and highly impactful features. - Establish scalable, efficient, automated processes for large scale data analyses, model benchmarking, model validation and model implementation. - Research and implement novel machine learning and statistical approaches. - Participate in strategic initiatives to employ the most recent advances in ML in a fast-paced, experimental environment. About the team Selling Partner Experience Science is a growing team of scientists, engineers and product leaders engaged in the research and development of the next generation of ML-driven technology to empower Amazon's Selling Partners to succeed. We draw from many science domains, from Natural Language Processing to Computer Vision to Optimization to Economics, to create solutions that seamlessly and automatically engage with Sellers, solve their problems, and help them grow. We are focused on building seller facing AI-powered tools using the latest science advancements to empower sellers to drive the growth of their business. We strive to radically simplify the seller experience, lowering the cognitive burden of selling on Amazon by making it easy to accomplish critical tasks such as launching new products, understanding and complying with Amazon’s policies and taking actions to grow their business.
US, WA, Seattle
Join us in the evolution of Amazon’s Seller business! The Selling Partner Growth organization is the growth and development engine for our Store. Partnering with business, product, and engineering, we catalyze SP growth with comprehensive and accurate data, unique insights, and actionable recommendations and collaborate with WW SP facing teams to drive adoption and create feedback loops. We strongly believe that any motivated SP should be able to grow their businesses and reach their full potential supported by Amazon tools and resources. We are looking for a Senior Applied Scientist to lead us to identify data-driven insight and opportunities to improve our SP growth strategy and drive new seller success. As a successful applied scientist on our talented team of scientists and engineers, you will solve complex problems to identify actionable opportunities, and collaborate with engineering, research, and business teams for future innovation. You need to have deep understanding on the business domain and have the ability to connect business with science. You are also strong in ML modeling and scientific foundation with the ability to collaborate with engineering to put models in production to answer specific business questions. You are an expert at synthesizing and communicating insights and recommendations to audiences of varying levels of technical sophistication. You will continue to contribute to the research community, by working with scientists across Amazon, as well as collaborating with academic researchers and publishing papers (www.aboutamazon.com/research). Key job responsibilities As a Sr. Applied Scientist in the team, you will: - Identify opportunities to improve SP growth and translate those opportunities into science problems via principled statistical solutions (e.g. ML, causal, RL). - Mentor and guide the applied scientists in our organization and hold us to a high standard of technical rigor and excellence in MLOps. - Design and lead roadmaps for complex science projects to help SP have a delightful selling experience while creating long term value for our shoppers. - Work with our engineering partners and draw upon your experience to meet latency and other system constraints. - Identify untapped, high-risk technical and scientific directions, and simulate new research directions that you will drive to completion and deliver. - Be responsible for communicating our science innovations to the broader internal & external scientific community.