Researchers audited 305 official Ollama GGUF tags across 15 code-capable model lines and 22 popular Hugging Face community files, discovering that public registries perform no functional testing of quantized LLM artifacts. A 15-task code screen flagged suspects, which were then evaluated on the full 164-task HumanEval+ benchmark. Seven official tags (2.3%) solved at most six tasks; all were low-bit quantizations of models with four billion parameters or fewer. Rebuilding five suspect files from original weights using identical toolchains and tensor mixes produced tensor-for-tensor matches, confirming the registry files were not corrupted but fundamentally non-functional. Crucially, adding an importance matrix during quantization — effectively calibrating the process — raised Qwen2.5-Coder-3B scores at three quantization levels from zero to 76–113 solved tasks, while the same uncalibrated recipe left Qwen2.5-Coder-1.5B functional, demonstrating that model scale interacts with quantization sensitivity. No false positives emerged: all 66 sampled tags passing the initial screen remained functional. The authors release the audit dataset and the quantcheck screening tool, recommending mandatory pre-publication screening of low-bit quantizations followed by calibration or withholding of failing artifacts.
Read original
huggingface/daily-papers