I spent two years making Tesla P100s and V100s not suck at LLMs. Today I'm releasing the engine, and I benchmarked it against llama.cpp, ik_llama.cpp and 1Cat vLLM on the same cards. Charts inside.

Article automatically generated from technical news.

Quick disclosure: this is my project. I built it because I have a rack of "obsolete" datacenter cards in a closet and I got tired of every new model needing a new set of flags to run properly on them. **What it is** PXA is a fork of ik_llama.cpp (which is a fork of llama.cpp) plus a vLLM plugin, built for cards with HBM2 and no tensor cores or DP4A: Tesla P100, V100, GTX 1080 Ti. It has its own quant format (PXQ, 2 to 6 bit, plus a mixed one that sizes a model to w