JevBench has been introduced as a reproducible benchmark specifically designed to evaluate typed decision models. The benchmark facilitates comparative analysis across various models, as demonstrated by its comprehensive performance leaderboard. According to the data, Jev 1.13.0 achieved the highest overall performance with a score of 74.4, followed by SemIf at 73.1. The "djev" model also demonstrated strong results, securing high rankings with scores such as 78.5 in one category and 81.7 in another. In contrast, lower-performing models like Certo v1 and Alibaba GTE Reranker ModernBERT-base scored 0.0 and 0.3 respectively, indicating the benchmark's ability to distinguish between highly effective decision models and those with minimal proficiency. The results show significant variance in model rankings across different testing metrics, with several models experiencing rank shifts compared to their primary JevBench Score. This benchmarking framework provides a standardized method for assessing how different architectures, ranging from large-scale models like Winnow-12B Q8 to smaller specialized versions, handle decision-making tasks.

Read original