PReMISE: Policy rubrics as measurement specifications for LLM judges

LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rubric asking for a response to be “helpful and factual” can reward polished answers that invent facts or violate user intent. We treat reusable rubrics as measurement specifications: changing the rubric changes the response quality measurement induced by a fixed judge. We introduce PREMISE, a framework that, given pairwise human-preference data, (i) discovers a policy-level rubric set, and (ii) audits any rubric set under LLM-judge use along four axes: structural adequacy, reliability, preference fit, and adversarial robustness. We evaluate PREMISE on four public human-preference corpora – UltraFeedback, Arena-Expert-5K, HelpSteer3, and HH-RLHF – across six rubric sources and three openweight judges spanning 32B to 1T parameters. Across rubric sources no raw source is simultaneously reliable, preference-predictive, and adversarially robust; and high inter-rater agreement does not imply low exploitability. PREMISE is the only rubric source to score non-trivially on applicability, specificity, and effective dimensionality simultaneously. We contribute two audit-targeted repair operations: preference-rank selection raises held-out crosssource, cross-judge mean accuracy on paired responses from 65.0% to 68.6%, competitive with the strongest rubric-discovery baselines and leading on two of three judges in our cross-judge sweep; reliability-constrained refinement reduces the rate at which exploit responses receive high scores from 46.4% to 36.0% with little change in inter-judge agreement (α=.531 → .519).