Unsloth released updated Qwen3.8-27B GGUF quantizations using the Dynamic V3.1 algorithm, improving average accuracy by 5-10% across benchmarks including Terminal Bench 2.1, DeepSWE v1.1, LiveCodeBench v6, GPQA Diamond, MMLU-Pro, and AIME25 at equivalent model sizes. The 27B-parameter dense model features a 256K context window (extendable to 1M via YaRN), vision and reasoning capabilities, and runs on 16-19GB unified memory for 4-bit quants. Dynamic V3.1 updates apply to most 2-bit through 6-bit quants, though 3-bit UD3 quants were retained as they matched V3.1 performance. The release includes developer role support for agentic tools like Codex, MTP-enabled fast inference, and improved nested object parsing for tool calling. Inference defaults differ for thinking mode (temperature=1.0, top_p=0.95, presence_penalty=0.0) and instruct mode (temperature=0.7, top_p=0.80, presence_penalty=1.5), with reasoning_effort toggles (xhigh, medium, low, none) for depth control. NVFP4 quants targeting Blackwell GPUs (RTX 50-series, B200, B300) achieve 1.5x speedup over BF16 with 92-97% top-1 accuracy recovery across code, chat, and multilingual domains, plus FP8 KV cache calibration for 2x context length. NVFP4 deployment requires vLLM (SGLang v0.5.19+ for MTP). The 2.4T-A95B MoE variant (95B active) is available as a 397GB IQ1_XXXS quant (1.1875 bpw, 91% smaller than 4.9TB FP), requiring ~450GB RAM and a specific llama.cpp branch.
Read original
hackernews