Flash Attention: exact attention without the N N memory blow-up
Flash Attention addresses the memory bottleneck in transformer models by eliminating the need to write the large attention matrix to memory. This technique allows for the computation of exact attention without the quadra…
→ View original source