Berth is a lightweight GPU resource manager written in Go that enables live-swapping of CUDA workloads by managing GPU access through leases, priorities, and a request queue. It coordinates VRAM sharing among applications like llama.cpp, ComfyUI, and Strata by waiting for tasks to finish, releasing memory, and handing control to the next queued application. This approach prevents VRAM contention and allows multiple AI workloads to share a single GPU sequentially.
Read original
reddit/r/LocalLLaMA