Kostya's Boring Codec World

In last post I wrote about how I’ve managed to reconstruct a recognizable picture for MVI1 codec. After I fixed the prediction code it started to work properly. Surprisingly, Treasure Quest game proved to be a source of MVI1 files in all formats (RGB, YUV422, YUV420, YUV410 and YUV4½0—the last one has one set of chroma samples per 4×4 block and is the most common MVI format in general). Additionally it has MVI1 samples with golden frame feature (I named it after a feature in a family of competing codecs that started with rather similar coding approach): frame 0 is two intra frames with the second frame serving as the background for the other frames; there is an additional map mode which tells that certain rectangles should be copied from the golden frame (instead of previous frame or filled with one colour). MVI2 seems to have an extension of that mode but I’ll see about it when I get to it (and if I obtain samples using that mode).

So, MVI2 next. Considering the number of extensions they added (and how they interfere with frame reconstruction) it’s probably not going to be easy but now I have a base to extend instead of blind guesses to make.

As I mentioned last month, I decided to reverse engineer Motion Pixel codecs and after a lot of failed attempts to make decoder work I’ve finally got something.

First of all, here are two frames from different videos.

MVITEST.AVI:

And a frame from SHOT09.AVI (from Apollo 18 game):

As you can see, the images are not perfect but recognizable already. And just last week it would’ve been a mess of colours with some barely recognizable distorted shapes.

The reason for this is that while MVI1 (and MVI2) is indeed based on stand-alone MVI format (I call it MVI0 for clarity), there are some nuances. On the first glance MVI0 and MVI1 are the same—all steps are the same—and indeed you can use the same code to decode data from either, but the reconstruction steps differ significantly.

Essentially there are four steps there: decode rectangles defining which parts of the frame will be left intact or filled with one colour, decode deltas used to reconstruct the rest of pixels, use those deltas to generate predictors for each line (where needed), use the rest of deltas to reconstruct the rest of pixels. Additionally MVI employs chroma subsampling mode so only one pixel in 1×1 to 4×4 block (depending on mode) has delta differences applied to chroma, all other pixels update only luma component. So if you don’t do it correctly you may end up applying e.g. deltas intended for luma to chroma components and vice versa. That’s what I got for a long time and could not understand why.

It turns out that vertical prediction has its pixel sampling at different position—or maybe it’s scrambled in the same way as line prediction. There for the most common mode (one set of chroma components per 4×4 block) each group of four lines is decoded in reverse order (i.e. 3, 2, 1, 0, 7, 6, 5, 4, …). For 2×2 block only lines in pairs are reversed. You can see artefacts of wrong prediction on Apollo frame.

Anyway, having a recognisable picture means that the hardest part (for MVI1) is done, so all is left now is to fix the remaining bugs, refactor the code and move to MVI2. There are other annoying things there but now I know how to deal with them.

BTW, if you’re curious why it takes so long, the problem is the binary specification being obfuscated to the point that Ghidra refuses to decompile most of MVI1 decoder functions and can’t do much about reconstruction functions since they’re a mess of spaghetti code (probably written in assembly language directly) so it’s more of a state machine than a decoding loop. And they abuse segment registers to access different parts of the context (and this must be the reason why it cannot work under OSes from this millennium). I got some progress when I resorted to debugging this mess by running MVI2 player OllyDbg under Win95 (emulated in DosBox-X) and constantly referring to Ghidra to see where to put breakpoint to trace a certain function. That process is definitely not fun for me but it gave results.

Overall, probably it could’ve gone better but I hope the rest won’t take as long.

Since I’ve done all improvements to NihAV that I wanted to do (beside vague “improve Indeo 3 encoder somehow” or “add some interesting format support”), I decided to look at the formats in my backlog and discovered that I have a QPEG player with a couple of DVC samples. Considering that I’ve REd their VfW codec over two decades ago, I had to look at this as well.

It turned out to be a straightforward format with static palette and video frames packed with moderately complex RLE (it has opcodes for run/copy/skip and literals). The most interesting thing there to me was that values without high bit set are treated as literals, or rather as indices in the remapping table (which is the first 128 bytes of a frame). Considering that low 20 colours of the palette seem to be unset, it makes some sense.

The hardest part was to read the binary specification. The executable uses Phar Lap 386 extender, so it’s actually stored in P3 format right after the loader. At least I have some experience with loading such formats when I messed with Linear eXecutable format before Ghidra plugins were available (and sometimes afterward as well, since e.g. neither of two known plugins managed to load Wing Nuts executable). Also I managed to spot that 0xE0 byte happens at the end of packed frame, so I guessed it was the end data marker and searched for the code using it as such. I’ve managed to locate four RLE decompression functions, all probably functionally identical, and after figuring out other details (like where remap table comes from) I ended up with the decoder that works on all four known samples just fine.

Overall, it’s nothing particularly complex but it was still nice to look at.

Since I have nothing better to do (beside some slight NihAV refactoring) and somebody told me about it, I decided to look at the format. Apparently back in the day The Multimedia Mike also attempted to research some information about it but I don’t think anything substantial came out from it.

Anyway, here’s what I everybody knows about it: as apparent from the name, it was developed by Johnson-Grace company, it combines a lot of different image compression methods and format—so apparently you can have slide show with an accompanying MIDI or speech, and it splits image into tiles and tries to compress them using whatever method fits best.

So, here are some additional details.

Audio codec is a rather common speech codec (LPC plus quality improvement post-filters) with one peculiarity: internally it decodes 16-bit samples yet outputs 8-bit PCM.

Slide show—I had just a cursory glance but it seems to mix various kinds of content with the slide show commands (like displaying next image) in the single file. Of course it’s last in my priorities list.

Image formats—now that’s where the real fun is. It handles about twenty different chunk types (even if most of them are useless and provide some image information at best) and recognizes (and skips) about the same amount too. I’m still struggling with the code but there seem to be three types of compression: LZ77-based lossless compression, lossy compression with the same coding for coefficients that probably uses wavelet coding, and another lossy compression (for palette-based images?). So far the only things I’m sure about is that it employs LZ77-based compression that reminds me of deflate with dynamic codebooks (but differs from it or DCL) and it seem to code signed coefficients while at it; the other thing is there are way too many functions for converting palette formats (usually between 24- and 32-bit RGB but quite often it’s between 32-bit RGB and 32-bit RGB in the same format but as an integer or an array of bytes).

In either case I’m in no hurry and can keep digging into it at my leisure.