博主雷达 / @tri_dao
@tri_dao
粉丝 4.6万关注 667最近同步 10/19 15:41雷达库已收录
首席科学家；普林斯顿助理教授 · Together AI
去 X 主页数据积累中
无数据
TPU 的瓶颈和 GPU 不一样：核心少，所以不用太操心并行；VMEM 大，所以 kernel 融合更容易；但向量单元吞吐低，所以任何非矩阵乘法的操作都要格外小心
Releasing FLAT: Flash Linear Attention on TPUs 🚀 Pallas training kernels for Linear Attention, Mamba-2 and GLA. On one TPU v6e, forward + backward is up to 3x faster than FLA on an H100 for GLA, and 1.5x faster for Mamba-2. 🧵
查看被引用的帖子SSM 有着漂亮的渐近复杂度，却一直错过 Transformer 免费获得的推理红利，比如投机解码和量化。结果发现，我们只是一直没缓存对的东西。FlashAttention、Mamba 扫描、分页 KV 背后都是同一套 IO 感知的技巧。很多大胜利并不是新架构，只是人们开始认真对待内存 IO 了。
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
查看被引用的帖子