Developer u/5_ChubbyCheekz23 released PXA 4.0, a fork of llama.cpp optimized for older NVIDIA Tesla P100 and V100 GPUs. The project achieves up to 3x faster decode and 8x faster prompt processing speeds compared to mainline llama.cpp on identical hardware. The optimization addresses the lack of tuned kernels for Pascal and Volta architectures and enables large MoE models to fit on these affordable, high-memory cards.
Read original
reddit/r/LocalLLM