arXiv (All)
open access
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B token
Conceptio Open Archive
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
This document is indexed with metadata only: full text is not available in the archive for this record. Open the official source ↗
Related documents
- Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training#1108716
- FLASH-MAXSIM: IO-Aware Fused Kernels for Late-Interaction Retrieval#1117440
- CVE-2026-89855#989059
- Bridging the Vendor Gap: Enabling AMD GPU Support for Awkward Array via ROCm/HIP for the HL-LHC Era#1051037
- Bridging the Vendor Gap: Enabling AMD GPU Support for Awkward Array via ROCm/HIP for the HL-LHC Era#1028652
Record · ID 1119345
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Bring this data into your agent: Connect via MCP or API · 20 free credits to start · no card required
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive