Small Language Models as Judges: A Survey
Anish Laddha, Nitesh Pradhan, Gaurav Srivastava
Department of Computer Science and Engineering, LNMIIT, India; Department of Computer Science, Virginia Tech, USA
Abstract
Automated evaluation of language model outputs has shifted from costly human annotation toward model-based judging, yet the field has relied mostly on large proprietary models whose cost, opacity, and data-privacy risks limit scalable deployment. This survey examines Small Language Models (SLMs, ≤14B parameters) as judges, synthesizing 105 works spanning the full pipeline from supervised fine-tuning and preference optimization through reinforcement learning with verifiable rewards to representation-based probing. We organize the field into five dimensions and distill five insights, each graded by the strength of its evidence: (I1) evaluation-specific training matters more than raw scale; (I2) reasoning tokens help only when they surface genuinely new discriminative signals; (I3) cross-family ensemble diversity decorrelates errors; (I4) debate value is bounded by information novelty, not round count; and (I5) SLM judge reliability tracks the verifiability gradient of the domain. In every surveyed paper that reports a trained judge next to its own base model under one protocol, evaluation-specific training improves the base model, with relative gains of 15–63%. Position-consistent accuracy protocols further show that much apparent judge performance is heuristic-driven, and that all judges, including frontier models, degrade sharply on hard evaluation pairs. Across surveyed studies, fine-tuned 3–8B judges match proprietary-model accuracy on standard benchmarks, and heterogeneous panels of smaller models reach comparable quality at over 7× lower API cost; this parity narrows in subjective domains that lack external verification. Small judges also make private, on-device evaluation practical for continuous Reinforcement Learning from Human Feedback (RLHF) loops. We close with a research agenda on calibration, adaptive inference, and generalization.