Bah! Late last night I thought about adding Triangular Dithering to Prometheus. This resulted in discovering that the rectangular dithering was wrong. It actually used quarter-rectangular rather than half-rectangular.
After adding triangle, the results still didn't match those of Sony SoundForge's dither. One problem with all of these dithering algorithms is the testing. I can output a stream of numbers as a sample, but we're testing in fractions of a sample, so most of the results are about statistical analysis.
My first job today was to create some test waves. I created a 1 sample 16-bit wav, then converted to 32-bit, and saved out 50%, 75% and 100% values as 32-bit test waves. The hard part is testing the resuling dithers.
For a 50% wav dithered at 16-bit, it should be an 1:1 mix of 0 and 1. This is easy to test; maximise/normalise the result and the average volume should be 50%.
For a 100% wave, it should be all 1 -perhaps-, though the rectangle may -just- touch 0 and 2. The average, however should be 1. Sony's Dither wasn't, for some reason. Mine varied between 0 and 2, producing a 50% average over a 28sec wave (over a million samples, a big enough test!)
For a 75% wave however, the results were harder to determine. Sony's dither, when maximised, created a wave with a 75% average. Mine produce an average of 37.4%. My floating-point results seemed correct. For 75%, the samples varied from -0.000007630 to 0.000053407 with 0.000022869 average; which is about 75% of one 16-bit sample.
What turned out to crucial was the 16-bit conversion. An important thing to note is that a 16-bit number can range from -32768 to +32767 but in 16-bit samples values should range -32767 to +32767; and this is the case in about half of (but not all) sample editors. It makes no sense to make the lower half of a sample have a different resolution to the top. When dealing with high-end audio, that technical distortion of the DC offset is not acceptable. Zero should be the centre of the wave.
Short to float conversion rounds down, and I forgot to add half-a-sample before doing this, so 0.9 rounded to 0, rather than 1. This makes all the difference when dealing with dither-level events.
I thought of the 75% result. We have three sample blocks A: -1 to 0; B: 0 to 1; C: 1 to 2. 25% of A is covered, 100% of B, and 50% of C. Now, that float conversion happens halfway though these blocks, so the resulting dither should create 3 units of sample 0; 4 of sample 1; and 2 of sample 2. This means that the average should be lower than 50%, so my 37% seemed correct...
More testing needed.
