I admit that I haven’t spent much time on writing encoder but I still have some progress to report.
VP6 is a rather simple codec and intra frames there have just three parts: frame header, model probabilities update and macroblock data (just block coefficients). Frame header has fixed entries (unless I do VP6 advanced profile, which is unlikely), model update can be avoided by sending lots of zero flags signalling there’s no update from the default, block coefficients are a bit trickier but essentially it’s just writing coefficients and zero runs.
As you still have to do a lot of the stuff like DC prediction (for removing predicted part instead of adding it), keeping models that are used for context-dependent coding and such, I simply reused a great deal of existing NihAV VP6 decoder data and code since it does that already.
So now I have a simple encoder that takes input frames, performs DCT, quantises blocks with the fixed quantiser and writes the data using the default model probabilities. After lots of debugging it seems to work as expected.
Here’s a list of troubles I had to deal with:
- first of all, it turned out that bool coder can have overflows after all so I had to deal with them (now it’s done a bit inefficiently but I hope to improve the implementation);
- I also forgot to write the tail value from bool coder properly at first so the last blocks were decoded wrong;
- there were some difficulties in block encoding coming from its context-dependent nature: DC context is dependent on DC value of top and left neighbour block after prediction, zero run uses context derived from its start position (so you have remember it too), plus you have to track when the coefficient should be coded using non-zero or zero-including tree;
- and the least logical thing of them all: frame header stores some of frame information as normal fields but frame dimensions are coded using bool coder for some unknown reason and if you treat them as ordinary bytes you’ll have both wrong dimensions and wrong bool coder internal state.
Anyway, the most important thing is that it works and produces valid frames. I still have to do a lot of code clean-up and fix some bugs too, but the most important thing is to calculate the optimal probabilities (and then rough them up to match the way they are stored or calculated—otherwise you’d have to transmit even more probabilities). This will require essentially duplicate code but with updating probability counters instead of coding.
And that’s not talking about multipart and Huffman encoding that may be done eventually (or not).
That makes it a long way to part 2 so don’t expect it soon (and inter frame encoding will be even more complex).