The PCST (Product Code Structured Transform) project investigated extreme low-bit compression of LLaMA-7B weights without retraining, distillation, or pruning. Utilizing a post-training mathematical approach involving Product Quantization (PQ), SVD residuals, and orthogonal gauge transformations, researchers developed the PCST v5 artifact. This 2.05 GiB representation employs a heterogeneous allocation strategy, assigning different bit budgets to various layers and matrix types based on sensitivity. While the method achieved a Top-10 overlap of 59.59% and a next-token accuracy of 43.56% on WikiText-2, it remains significantly outperformed by the 3.07 GiB Q3_K_M model, which reached a perplexity of 7.36 compared to PCST's 15.56. A notable success was the implementation of a rank-32 low-rank bridge between PCST hidden states and the LM Head, which improved Top-10 overlap by 1.706 percentage points and increased logit cosine similarity from 0.8538 to 0.8823. However, the study concluded that reducing local weight Mean Squared Error (MSE) frequently fails to improve end-to-end token ranking, suggesting that effective extreme compression requires network-wide error shaping rather than isolated matrix-level optimizations.

Read original