Japan Open Data Lab released ja-injection-guard-v1, a 130M-parameter text classifier (sbintuitions/modernbert-ja-130m base) that screens Japanese and English inputs for prompt-injection attempts before they reach an LLM. Trained for one epoch on 45,000 items — combining llmail-inject-challenge, browsesafe-bench, and synthetic Japanese/English attacks with long-form benign documents — the model was evaluated on a sealed 2,500-item test set with a threshold (0.394) fixed on development data. At that threshold it achieves 93.0% attack detection, a 0.44% false-positive rate on benign text, and F1 0.9599. Detection varies by attack type: instruction override and fake conversation history reach 100%, indirect injection 96.7%, role hijack 94.7%, obfuscation 89.3%, while extraction attacks lag at 77.3%. False positives are 0.0000 on hard benign text, 0.67% on long documents, and 0.60% on ordinary benign text. External validation shows 0.0000 false positives on XSTest safe (250 items) and 9.41% on NotInject (170 items), revealing multilingual sensitivity beyond Japanese/English. CPU inference averages 104 ms (median 50 ms) on a Core i7-11700F; an ONNX fp32 build matches PyTorch predictions exactly. Limitations include degraded performance on long inputs (>512 chars, 2.11% false positives), no INT8 quantization (raised false positives), and the explicit caveat that this screen is one defense-in-depth layer, not a guarantee. Apache 2.0 licensed, available on Hugging Face and Kaggle.

Read original