Researchers Obed Junias and Maria Leonor Pacheco address a specific failure mode in Large Language Models (LLMs) where models struggle to process compound answer options containing logical operators like AND, OR, and NEITHER/NOR, even when individual atomic judgments are accurate. To mitigate this, the authors propose a framework that avoids presenting compound options directly to the model. Instead, the method decomposes each option into its constituent atomic answers and uses contrastive hypothesis scoring to evaluate each atom independently. These calibrated scores are then integrated using an operator-constrained integer linear program to derive a final logical composition. The effectiveness of this approach was tested on LOGICAL-COMMONSENSEQA and a newly introduced reading-comprehension benchmark, LOGICAL-SATA. Experimental results demonstrate significant performance improvements: Macro-F1 scores rose from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA. Notably, the framework yielded the most substantial performance gains when handling NEITHER/NOR operations, proving that structured decomposition and mathematical composition can overcome the reasoning limitations inherent in direct compound option processing.

Read original