Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
arXiv cs.AI · 7 Nov 2026, 04:29 AEST
This paper describes an approach to estimate the correctness of LLM-generated COBOL summaries without human-authored references. The methodology utilizes inter-model agreement and a LLM evaluator to provide structured judgments on summary quality.
Key facts
- This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references.
- The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics.
- We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.
Read aloud
This paper describes an approach to estimate the correctness of L L M-generated COBOL summaries without human-authored references. The methodology utilizes inter-model agreement and a L L M evaluator to provide structured judgments on summary quality. This paper describes an approach and use case to estimate the correctness of L L M-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an L L M-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with i controlled summary generation from diverse models; ii a L L M evaluator that compares pairs of summaries and emits structured judgments; and iii a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.
Sources in this cluster
- Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
arXiv cs.AI · 7 Nov 2026, 04:29 AEST