Kostya's Boring Codec World

As I’m somewhat tired with na_game_tool development (for now), so I’m just picking bits for the release goal to do that and switch to something else (I’ve found a couple of interesting MOV files to analyse, maybe I can improve my Indeo 3 encoder and such). I’ve added support for HNM5 and HNM6 formats, reached the self-imposed limit of at least twelve original decoders and essentially I need to add 7th Level archive format and that’s it. Meanwhile I can talk about the formats I added support for, I want to add support for (but not right now) and the formats I’m hesitating to add support for.

So, supported formats first.

There’s GRN format used in Genesia game. Frames are stored raw, compressed with RLE (which looks a lot like FLI delta RLE compression but working on pixel pairs) and the data may optionally be compressed with LZSS afterwards. And if would’ve not been a format from French company without some weirdness. In this case audio stream data is interleaved with video in the simplest way: first you have 2kB header, then a certain number of 2kB sectors with initial audio data (the number is in the header), then it is 8kB of video stream followed by 2kB of audio followed by 8kB of video stream etc etc. So video frame data may suddenly have 2kB of audio inside it. Of course it was easily solved with a custom de-interleaver but you’d expect it to see in industry-standard applications and not in a game from early 1990s.

And there’s MGIF from Gates of Skeldal. Despite the name it was only inspired by GIF in the sense it uses similar LZW compression scheme but not the format (unlike HAF). Another curious detail that I haven’t seen elsewhere is using a simple pre-processing: bytes are coded as a difference to the previous ones. What made it annoying is that this prediction resets when LZW decoder resets (i.e. when the “reset dictionary” code is encountered), so you have to implement it inside LZW decompressor instead of making a pass afterwards. Still, it gets a point for the originality.

Finally there is Celestial Impact intro.dxv. This file employs RLE to compress not merely image but its palette and some additional data (it did not look like a sound to me and I have no idea what it is) as a single array.

Now for the formats I want to support in some (distant) future. Beside the usual “whatever I can find” it definitely includes AVI and SMS—two Smacker-based formats. Maybe I can implement some Cinepak-based console formats while at it.

And speaking of console formats, here’s a fun format called DDV and used in Oddworld games (or maybe just one of them). There’s a reverse-engineered implementation of the decoder and by the look of it this format encapsulates MDEC—which makes it a perfect candidate for librempeg. It has support for some MDEC-based formats already after all.

That’s it, hopefully the next post will be about the release already.

I’ve added another bunch of formats to na_game_tool, including both old video formats and game archive support to extract some of those (and other) formats from.

For example, I’ve finally added an extractor for Conquest Earth WAD with its 16-bit FLH (a variant of FLI with RNC compression; it has been supported since version 0.2.0) but there are other variants that justify this post title:

• Alien Virus animation—almost standard FLI with changed tag and some jitter in subchunk sizes (i.e. they may be a byte too short or a byte too long compared to the chunk size);
• Bureau 13—here it is a super-format with chunks comprising FLI headers, FLI chunks or PCM audio—and it may be several FLIs with different resolution too. And it is put into its own archive format (which I also added an extractor for);
• C13 for Hammer of the Gods—at first I thought it’s just yet another hack of the format but after looking closer at the data I realised it’s merely LZSS-compressed. Essentially I just hacked my FLI decoder to decompress data first and operate on decoded data instead of file in this case;
• Stargunner FLC—like other files in the game archive it was compressed using byte-pair encoding (and it may be the only case of such method being used for compression in the wild). In this case file is split into small chunks and unused byte values are used to code pairs (of used byte values or other pairs). Adding support for it was as trivial as in the previous case, but the fun thing here is that after I figured out the decompression algorithm I found out that it’s been known and supported by various extraction tools, it’s just discmaster picked up the laziest one.

Beside that I’ve REd HUF format from Johnny Bazookatone which employs RLE compression and then further compresses video frame data with global Huffman codes. And I’ve also added support for Goosebumps: Escape from Horrorland archive among other things. The curious thing about it is that such archives contain .gvd files which are essentially remuxed AVIs, so my plug-in reconstructs AVI from them by default (but this can be disabled if you prefer the original files instead). Almost all of the files use Indeo 3 but a couple of them seem to use their own codec with no decoder available so here’s that.

In general I still have at least four original formats to add (plus HNM6 and some extractors) but there seems to be enough candidate games to RE for it to be feasible.

P.S. I also looked at SMV format from AGON: The Lost Sword of Toledo but after looking at the binary specification and discovering MPEG-4 ASP decoder there (also if you remove the SMV header it can be played as an elementary MPEG-4 ASP stream) I lost any interest. Maybe Paul will have some interest supporting it in librempeg, maybe not—and I definitely don’t want to mess with such format (the same applies to KSV as well).

Last time I talked about MVI formats, mentioning that I had one more equally German MVI format. Well, the official specification uses CauseWay DOS extender which compresses executables (and I still haven’t found a way to make DosBox debugger dump loaded 32-bit segments, otherwise I would’ve REd a bunch of Psygnosis formats too). Luckily the demo version did not use it and I was able to find how it is coded. Apparently they employed some non-standard LZW which for some reason shuffled low codes, so 0/1/2 are used as special signals—dictionary bump (followed by byte aligning for some reason; other implementations bump dictionary size implicitly and do not insert bits), dictionary reset and EOF correspondingly—while codes 3..257 are used to code bytes. Beside that it’s nothing special: intra frames are (optionally) LZW-compressed, inter frames consist of pixels and mask telling where on frame to update those pixels (both parts with optional LZW compression).

Speaking of LZW, there was this FLK format (apparently Italian) that also employed LZW to compress pixel data along with the command stream with rather simple commands like “repeat pixel N times, skip next M pixels, copy following P pixels” and additionally “restore Q pixels of the background”. As you can guess, the format is used for animations overlaid on something else in addition to video clips. In case of the latter command stream may be absent and you simply put decompressed pixels into a new frame.

Other codecs are Ascon SKS (which is essentially a collection of JPEGs with swapped chroma planes; I did it mostly as a test of JPEG decoder module that I plan to use in other decoders) and Interactive Pictures VID (which I encountered in .evd, .fvd and .gvd files—for files with English version of the video, French version, and general version without speech). This one is curious not only because its binary specification for some reason contains compression for it (which I encountered first) but for the format features itself. Images are split into 8×8 blocks that may be coded raw, with 2-16 colours and a pattern, and a custom-scan RLE I remember seeing in Bink (and XCF but like Bink there are 16 patterns here as well and not just four). Probably nobody cares about it but it was fun to discover.

In addition to that I’ve implemented support for a couple of well-known formats, namely HNM5 (aka UBB2—UBB is apparently the same but lacks headers) and RoQ. The former is a typical French format (you can tell it by the fact it uses crazy motion compensation modes with mirroring and transposing), I implemented it mostly for completeness sake (so I have support for all HNM flavours in my tool, all is left is HNM6). The latter has the same rationale as when I looked at it four years ago: its support in libavformat/libavcodec sucks i.e. it’s adequate for videos intended for id Tech 4 engine but not for the original Trilobyte games (not handling JPEG-compressed frames and not descending into 0x1030 chunk are two most glaring deficiencies). Plus there is a can of worms related to the fact that audio in Trilobyte games uses unsigned predictor while id Tech games use signed predictor and to the fact that for some files you need to scale motion vector twice (and for some files it should be done on a picture scaled twice vertically). I try my best to detect such situations but they do not offer work correctly (and if you wonder, in games engine may have a hard-coded list of files that should be treated differently).

And that is actually not all! I’m still working on supporting some game archives as well (for example, the majority of RoQ files I encountered were hidden in .gjd archives, same as VDX files). One interesting example is Escal compressor. One funny aspect of it is that it is employed in Pumuckl Klabauterjagd, a game for about 8-year olds, while the rest of titles using it are definitely 18+. From technical point it’s interesting as it seems to be inspired by the dynamic variant of deflate with some simplifications: instead of blocks you have codebook definition valid only for the next specified amount of symbols (which are still combined literals and copy symbols, with their codebook coded using code lengths packed with another codebook). Overall it’s distinct enough but the source of inspiration is obvious too.

Overall it means I have only four more original decoders to create plus some of the archive formats to support (like Coktel Vision STK/STK2 as well as the previously mentioned extractor/converter for Monty Python & the Quest for Holy Grail). Probably it will be the last release too as I’m not sure I’ll have enough game formats for another release. Well, there’s also na_eofdec which should get more Amiga formats before 0.1.0 release. And in theory there are console formats if Paul has not written decoders for them all…