Clive Chan
@itsclivetime · Nov 10, 2025
Floats are complex, but they’re often cheaper in silicon! They’re made of two *smaller* integers (exp, mant), and E4M3 FP8 mul is dominated by the 4b mantissa mul which is far smaller than an INT8 mul.
(INT8 is also a worse fit for typical weight/act distributions.)
Elon Musk@elonmusk· Nov 9, 2025@CernBasher As the number of bits drops, the difference between floating point and integer decreases until they are the same thing at 1 bit.
“Floating point” is not real. It is emulated with 2 integers and a lot of complexity.
Elon Musk
@elonmusk
AI is moving to primarily 4 bit weights. Int4 for Tesla inference.
Like a physical address, which has state, city and street. If you already know the state and city, only street need be specified.
You still get the precision you need with far fewer bits.
02:10 PM · November 10, 2025 · 29.6K views
59
52
781