DePEFT: Crowdsourcing LLM training/fine-tuning without a centralized cloud.

Article automatically generated from technical news.

A while ago, I successfully trained a tiny 80M parameter language model from scratch using the TinyStories dataset. It worked surprisingly well and could generate coherent little stories! Inspired by that success, I wanted to scale up and train a larger model capable of coding. That’s when reality hit me. I completely underestimated the massive compute and financial cost required to train larger LLMs. My single GPU choked, and renting cloud compute was way out of my budget. It got me thinki