Posit floating point numbers: thin triangles and other tricks (2019)

One of the creators of posits here (I came up with the name and i think ES is my idea, did the first full soft versions in julia, and designed the first circuits, including a cool optimization for addition). my personal stance is that posits are not great for scientific work precisely because of the difficulties with actually solving error propagation. Hopefully i can give a bit more measured insights into why the "parlor tricks" appear in the posits context.

John's background is in scientific compute/HPC and he previously advocated for using unums (which do fully track errors) and there is a version of posits (called valids) which do track errors, encouraging the user to combine with other techniques to cut the error bounds using invariants, but that requires algorithmic shift. Alas, a lot of examples were lifted from the unums book and sort of square peg/round holed into posits. you can see an example of algorithmic shift in the demo of matrix multiplication in the stanford talk (that demo is me; linked in OP).

as for me, i was much more interested in lower bit representation for ml applications where you ~don't care about error propagation. this also appears in the talk.

as it wound up, Facebook took some interest in it for AI but they nih'd it and redid the mantissa as logarithmic (which i think was a mistake).

and anyway redoing your silicon it turns out to be a pain in the ass (quires only make sense in the burn-the-existing-world perspective and are not so bad for training pipelines, where iirc kronecker product dominates), but the addition operation takes up quite a bit more floorspace, and just quantizing to int4 is with grouped scaling factors is easier with existing gpu pipelines, even custom hardware too.

fun side fact: Positron.ai, was so-named on the off chance that using posits makes sense (you can see the through line to science fiction that i was attempting to manifest when i came up with the name)

A while ago I built an interactive tool to display posits (Also IEEE floats etc.): https://mserdarsanli.github.io/FloatInfo/

It is hard to understand at first but after playing with this a bit it will make sense. As with everything, there are trade offs compared to IEEE floats, but having more precision when numbers are close to 1 is pretty nice.

Condensed IEEE-like formats cheat sheet I threw together and tested:

https://pastebin.com/aYwiVNcA

The example where computing an expression with posits has much better accuracy than when computing with IEEE FP32 is extremely misleading.

Regardless whether you use 32-bit posits or IEEE FP32, you can represent only the same count of numbers, i.e. of points on the "real" numbers axis.

When choosing a representation format, you cannot change the number of representable points, you can just choose to distribute the points in different places.

The IEEE FP32 format distributes the points so that the relative rounding error is approximately constant over the entire range.

Posits crowd the points into the segment close to zero, obtaining there a better rounding error, with the price that the segments distant to zero have very rare points, i.e. very high rounding errors.

Posits behave pretty much like a fixed-point format that has gradual overflow instead of a sharp cut-off. For big numbers you do not get an overflow exception that would stop the computation, but the accuracy of the results becomes very bad. For small numbers the accuracy is good, but not as good as for a fixed-point number, because some bit patterns must be reserved for representing the big numbers, in order to avoid overflow.

The example that demonstrates better accuracy for posits is manufactured by choosing values in the range where posits have better accuracy. It is trivial to manufacture an almost identical example where posits have worse accuracy, by choosing values in an interval where FP32 has better accuracy.

There are indeed problems where posits can outperform IEEE FP32, but it is quite difficult to predict which are those problems, because for a complex problem it can be very difficult to predict which will be the ranges for the intermediate results. This is the very reason why floating-point numbers are preferred over fixed-point numbers, to avoid the necessity of such analyses.

While for IEEE formats it is possible to make estimates of the relative errors of the results of a long computation, due to the guaranteed bounds for the relative error of each operation, that is pretty much impossible for posits, where the relative error is a function of the values of the operands, so you cannot estimate it without actually doing the computation.

For scientific and technical computations, posits are pretty much useless, because those have very wide value ranges for their data, also because those computations need error estimations, and also because posits can have significant advantages only for small number formats, of 32 bits or less, while those computations need mostly 64-bit numbers or even bigger.

Nevertheless, for special problems that are very well characterized, i.e. you know with certainty some narrow ranges for the values of the input data and of the intermediate results, posits could get much more accuracy than IEEE FP32, but they could have good performance only if they were implemented in hardware.

Great dive! I'm very interested in posits (and ieee float replacements in general) and never read this post before. Tons of insightful points.

This seems to be related to the "type III unum":

https://en.wikipedia.org/wiki/Unum_(number_format)#Posit_(Ty...

[flagged]