Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage.
To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors.
Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00–20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.
A rubric may ask a medical assistant to request the patient's location before advising on post-stent checkups. A response that only says regional guidance differs can still receive full credit under an underspecified criterion, despite never asking for the location or giving a checkup frequency. MetaRubric targets this mismatch between credited behavior and content actually present.
In HealthBench training with Qwen3-8B, using either GPT-4o-mini or GPT-5.4-mini as the reward judge, training reward increased while a separate test-set panel score declined. The panel used GPT-5.5, Gemini 3.5 Flash, and DeepSeek-V4-Pro, counting a criterion as satisfied only with unanimous agreement. An independent audit found that credit for fully omitted requirements persisted during training.
For 1,000 policy rollouts judged correct during training, the paper compares deleting required content with deleting generic statements while preserving the required content. Award retention measures the share of initially credited target criteria still credited after each edit.
| Reward judge | Required content deleted | Generic statements deleted |
|---|---|---|
| GPT-4o-mini | 82.0% | 93.1% |
| GPT-5.4-mini | 69.8% | 95.0% |
Most target awards survive deletion of the content needed to satisfy the criterion. Since GRPO compares rewards within each rollout group, such awards can change which responses receive positive advantage and are reinforced.
Each training stage holds the rubric fixed while the policy learns, then adapts criterion weights and descriptions from the errors in that stage's responses. Original prompts are paired with counterfactual prompts that change one task-relevant fact; each response is scored against the rubric for its own prompt.
For each criterion, a judge checks satisfaction, coverage, and support. Its credit is capped by the weakest of these three scores. A response-level support cap and a quality factor further constrain the rubric reward. A separate Qwen3-1.7B reader answers fixed multiple-choice questions using only the response, checking whether task facts can be recovered from it.
Criteria share weight adjustments by severity and correspondence type. Groups missed more often gain relative weight, while severity order and bounded adjustments preserve priorities. A proposer may clarify one criterion description at a stage boundary. It is accepted only after held-out judgment agreement improves beyond the threshold and a separate review confirms the initial requirement is preserved.
MetaRubric improves over static-judge GRPO in all 21 model-family and metric comparisons. It has the highest score among training methods in 19 of 21 comparisons; Dr. GRPO leads on HealthBench-Hard for Qwen3-4B and MMOral-OPG for Gemma-e2b. PubMedQA gains over GRPO are 6.00, 6.80, and 20.40 percentage points for Qwen3-4B, Qwen3-8B, and Gemma-e2b. The three open-ended benchmark gains range from 1.29 to 3.82 points.
PubMedQA uses answer accuracy. The other benchmarks use HealthBench-Hard accuracy-axis score, MMOral-X mean score, and MMOral-OPG overall score. GPT-5.5 is the strong judge for model-based test-set evaluation. Auxiliary QA accuracy measures exact option matching by the Qwen3-1.7B reader, with invalid outputs counted incorrect.
| Model / training method | PubMedQA | HealthBench-Hard | MMOral-X | MMOral-OPG | |||
|---|---|---|---|---|---|---|---|
| Acc. | Acc. | Aux. Acc. | Avg. | Aux. Acc. | Avg. | Aux. Acc. | |
| Reference Models | |||||||
| LLaVA-OneVision | 70.40 | 0.00 | 6.92 | 6.31 | 32.75 | 24.86 | 36.49 |
| InternVL3.5-8B-Instruct | 69.80 | 1.85 | 10.20 | 3.13 | 24.93 | 31.49 | 48.32 |
| MiniCPM-V 4.5 | 54.20 | 1.59 | 10.30 | 3.12 | 20.19 | 26.52 | 35.17 |
| MedGemma 1.5-4B | 55.20 | 4.88 | 18.24 | 5.77 | 25.82 | 24.46 | 32.21 |
| Qwen3-4B family | |||||||
| Baseline | 52.60 | 8.83 | 29.46 | 3.38 | 27.34 | 20.59 | 34.45 |
| + GRPO | 72.40 | 10.56 | 31.09 | 5.72 | 34.66 | 23.18 | 37.21 |
| + DAPO | 73.80 | 11.74 | 33.57 | 7.12 | 38.43 | 24.61 | 39.14 |
| + Dr. GRPO | 76.60 | 13.28 | 34.56 | 6.94 | 38.24 | 25.72 | 40.37 |
| + GSPO | 74.60 | 11.38 | 33.11 | 6.42 | 36.88 | 24.08 | 38.63 |
| + MetaRubric (ours) | 78.40 | 13.02 | 34.93 | 7.73 | 39.37 | 26.35 | 41.18 |
| Qwen3-8B family | |||||||
| Baseline | 44.20 | 4.24 | 30.19 | 3.50 | 28.24 | 26.28 | 36.29 |
| + GRPO | 70.00 | 7.15 | 32.86 | 7.08 | 35.41 | 27.35 | 38.02 |
| + DAPO | 76.60 | 8.64 | 34.52 | 7.86 | 37.20 | 28.44 | 39.35 |
| + Dr. GRPO | 76.40 | 9.72 | 35.07 | 8.58 | 39.61 | 29.63 | 41.08 |
| + GSPO | 76.20 | 8.18 | 33.28 | 7.52 | 36.69 | 28.06 | 38.94 |
| + MetaRubric (ours) | 76.80 | 10.34 | 36.14 | 9.21 | 41.09 | 30.44 | 42.61 |
| Gemma-e2b | |||||||
| Baseline | 48.00 | 11.23 | 32.74 | 3.69 | 30.17 | 32.77 | 34.86 |
| + GRPO | 51.60 | 13.07 | 33.62 | 4.74 | 33.82 | 31.28 | 37.00 |
| + DAPO | 67.80 | 13.82 | 34.66 | 5.82 | 38.04 | 33.64 | 38.53 |
| + Dr. GRPO | 70.80 | 14.94 | 35.92 | 5.71 | 37.61 | 35.42 | 39.76 |
| + GSPO | 69.60 | 13.34 | 34.08 | 5.07 | 35.43 | 34.58 | 38.12 |
| + MetaRubric (ours) | 72.00 | 15.68 | 36.85 | 6.03 | 38.70 | 35.10 | 40.57 |
Ablations with the Qwen3-4B family show the largest drop when both rubric descriptions and weights are frozen: 3.17 points on HealthBench-Hard and 4.40 on MMOral-OPG. Removing evidence checks lowers scores by 1.22 and 2.59 points, even with auxiliary QA and paired sampling retained. All variants keep the quality multiplier, auxiliary QA reward, and paired sampling.
| Dataset | Total | Revised | Avg. words before | Avg. words after |
|---|---|---|---|---|
| HealthBench | 10,448 | 4,727 (45.2%) | 40.8 | 46.3 |
| PubMedQA | 2,949 | 879 (29.8%) | 14.3 | 18.2 |
| MMOral-RL | 8,672 | 5,259 (60.6%) | 16.9 | 22.5 |
| Training rubric | HealthBench-Hard accuracy | MMOral-OPG overall |
|---|---|---|
| Baseline | 8.83 | 20.59 |
| Final rubric | 9.04 | 21.78 |
| MetaRubric | 13.02 | 26.35 |
The final rubric alone recovers only 5.0% of the HealthBench-Hard gain and 20.7% of the MMOral-OPG gain over baseline in these runs. The sequence of rubric updates contributes beyond the final rubric text.
For a PubMedQA prostate-bed motion question, the example aligns an abstract finding, a response, and a criterion before and after revision. The illustrative weight update moves an evidence criterion from +6 to +5 and an overstatement penalty from −6 to −8, giving more relative emphasis to the observed overstatement errors.
PubMedQA uses the non-test part of the expert-labeled PQA-L subset for training and the official 500-question test set for evaluation. HealthBench training and validation use conversations outside HealthBench-Hard; the Hard subset has 1,000 test conversations. MMOral-X and MMOral-OPG share the 980-example MMOral-RL training set; their test sets contain 300 and 578 questions.
Each training configuration uses one seed, so the reported scores do not quantify variation across runs. The supplied HealthBench Qwen3-4B configuration uses eight rollouts per prompt, sixteen responses per original–counterfactual group, rubric and QA reward coefficients of 1.0 and 0.3, and a revision threshold strictly above 0.02 on 20 validation prompt pairs.
The reader receives a model response plus the question and four options, then returns one letter. Questions are built from benchmark references and annotations; model responses are withheld during construction. HealthBench-Hard evaluation questions undergo content review without a mandatory question-only screening pass. Each MMOral evaluation bank retains all constructed questions once its valid-wrong screening rate exceeds 90%, including reader-correct items. PubMedQA instead uses its yes/no/maybe reference label directly.
| Benchmark | Source entries | Covered entries | Questions | Questions / entry |
|---|---|---|---|---|
| HealthBench-Hard | 1,000 | 965 | 4,117 | 4.12 |
| MMOral-X | 300 | 300 | 828 | 2.76 |
| MMOral-OPG | 578 | 578 | 981 | 1.70 |
Source request: “mention we only recommend a re-check if fever returns or symptoms worsen or fail to improve within another 2-3 days.”
Question: What specific follow-up timing should the rewritten message say for lack of improvement?
Correct answer: A
Source reference: “Endodontic treatments are noted on teeth 13, 23, 24, 26, and 45.”
Question: Which teeth have endodontic treatments?
Correct answer: B
Source reference: “Four wisdom teeth are detected: #18, #28, #38, and #48.”
Question: How many wisdom teeth are detected in the radiograph, and which teeth are they?
Correct answer: A
If you find MetaRubric useful, please cite:
@misc{fan2026metarubric,
title = {MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning},
author = {Fan, Yuxuan and Yoon, Jaehong},
year = {2026},
eprint = {2610.02824},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.02824}
}