Marin is an open-source research program and software platform designed for the end-to-end development of foundation models, encompassing data curation, tokenization, pretraining, and evaluation. The framework utilizes a topological execution model, similar to a Makefile, where experiment steps are executed in dependency-ordered sequences. A primary technical focus of the project is the development of a massive Mixture-of-Experts (MoE) model, targeting 5e24 training FLOPs and over 500 billion total parameters. To support efficient scaling, the team introduced Delphi, an open scaling suite that utilizes the Google TPU Research Cloud to map compute budgets to model configurations. Delphi provides a methodology to extrapolate model performance from smaller scales (3e18 FLOPs) to significantly larger ones (1e23 FLOPs), inspired by the Pythia architecture. Marin's capabilities extend beyond large language models to include audio-text, DNA, and protein modeling. Notable achievements include training an 8B parameter model that outperformed Llama 3.1 8B on base-model benchmarks and validating MoE quantile balancing at a 32B-A5B scale (1e22 FLOPs). The project emphasizes radical transparency by releasing all training pipelines, recipe code, and experiment checkpoints on Hugging Face.

Read original