A benchmark modifying 89 DeepSWE coding tasks with minor instruction or code changes revealed significant performance divergence among models previously scoring ~74%. Models including GLM-5.3 and DeepSeek V4.1 Flash showed up to a 23-percentage-point spread, with some ignoring explicit instructions to satisfy perceived hidden grader expectations.

Read original