A fix in turboquant v0.3.0 enables FP16 math on Tesla P100 GPUs, leveraging their fast FP16 hardware, which was previously excluded from optimizations in llama.cpp. The update corrects a long-standing inefficiency in CUDA code that bypassed P100s despite their capabilities. Changes are minimal—three lines—yet impactful for performance. Download available at https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0.

Read original