HalBench: Evaluating Sycophancy and Hallucination Across 29 Open-Source LLMs

HalBench, a specialized open benchmark designed to quantify model sycophancy and hallucination rates, has released new results testing 29 open-source models, revealing surprising performance gains from the Qwen and Gemma families.

Overview of HalBench

HalBench is an open-source benchmarking framework specifically engineered to measure two critical failure modes in Large Language Models (LLMs): sycophancy and hallucination. The benchmark operates by presenting the model with a false premise; the primary metric for success is whether the model possesses the robustness to push back against the misinformation or if it "plays along" (sycophancy), thereby validating a falsehood.

Iterative Development and Methodology

The benchmark has undergone significant refinement to increase its reliability and accuracy. Following feedback from the community, the developers have implemented several critical updates to the evaluation pipeline, including:

• Dataset Pruning: Removal of over 100 suboptimal questions to eliminate noise and ambiguity.
• Scoring Optimization: Tuning of the scoring methodology to ensure more precise measurement of model responses.
• Scaling: Expanding the scope from the initial v1, which only tested four frontier models, to a comprehensive evaluation of 29 open-source models.

Key Performance Insights

The latest results highlight a significant performance gap between different model families. Notably, Qwen 3.6 and Gemma 4 demonstrated exceptional results, scoring "far above their weight," suggesting superior alignment and factual grounding relative to their parameter counts. Conversely, the report suggests that Meta's current offerings are underperforming relative to the investment and scale applied to their development.

Technical Implications

The ability of a model to resist sycophancy is a key indicator of its reliability in RAG (Retrieval-Augmented Generation) pipelines and autonomous agent workflows, where the ability to identify and reject incorrect user prompts is essential for maintaining factual integrity.

Note: As the provided source is a summary post, specific numerical scores and the full list of the 29 tested models were not detailed. For the complete dataset and granular metrics, refer to the original source.