Researchers introduce Kierston, a city-scale simulation engine modeling approximately 125,000 residents of Allentown, Pennsylvania, and KierstonBench, a benchmark evaluating large language models (LLMs) on crime investigation and surveillance tasks. The simulation generates daily activities and distributes resulting telecommunications, financial, administrative, and license-plate records across sources representing separate providers, all derived from a common behavioral timeline. Models interact via a restricted records interface, linking observations across sources without accessing the underlying simulation state. KierstonBench comprises 32 task families spanning record linkage, routine reconstruction, incident investigation, and assessment of deceptive accounts. Evaluation of sixteen LLMs revealed most models failed to conduct valid records investigations, frequently submitting identifications unsupported by the evidence. A small subset of frontier models solved the majority of tasks and appropriately abstained when evidence was insufficient; however, these models remained susceptible to common deceptions and false assumptions, and none achieved performance equivalent to a human investigator.
Read original
huggingface/daily-papers