The researchers introduce onPanda, an interactive annotation framework designed to streamline the creation of alignment data for Large Language Models (LLMs) and autonomous agents. Unlike traditional manual post-editing, onPanda utilizes a "locate-correct-continue" loop based on token-level correction. During this process, annotators identify the first erroneous token in a model-generated response and rectify it by either selecting a replacement from the model's existing candidate tokens or utilizing free-form text editing. Once corrected, the system truncates the subsequent sequence and re-triggers generation from the new prefix, ensuring the resulting data remains consistent with the model's original sampling distribution. This method is particularly effective for generating high-quality on-policy supervised fine-tuning (SFT) and preference datasets, as it provides fine-grained supervision with precise positional markers for positive and negative token pairs. A controlled study demonstrates the efficiency of this approach, showing a 52% reduction in median annotation time compared to standard manual editing methods. Additionally, the system supports interactive trajectory annotation by integrating with external tools and harnesses. To support further research, the authors released Panda-CVL, a dataset annotated via onPanda, alongside a dedicated benchmark for evaluating token-level correction performance.
Read original
huggingface/daily-papers