SwarmEvolve introduces a latent-space evolutionary framework where hundreds of low-cost agents per iteration operate as swarms, reasoning without text decoding and communicating via Key-Value (KV) cache passing. Each swarm explores a distinct problem perspective, while a novelty filter maintains population diversity by measuring distances between agent hidden state vectors. Crossover occurs naturally in latent space as a summarizer agent processes the combined KV caches of all members. Evaluated on 21 mathematical discovery and scientific computing problems, SwarmEvolve achieves the highest average score across benchmarks, reaching optimal results in a median of 3 to 8 minutes compared to 60 minutes required by state-of-the-art baselines. Notably, using the Qwen3 8B model, SwarmEvolve improves upon solutions originally discovered by AlphaEvolve with frontier LLMs on several tasks. The approach demonstrates that latent-space agent swarms enable significantly faster and more cost-effective program search for scientific optimization.