Kostya's Boring Codec World

While discmaster is stagnating (you know whom to thank for the shortages of HDDs as well as RAM), I still look through stuff there in hopes to find something interesting. Occasionally I manage to stumble upon something special indeed.

This time it was navigable movies bundled with QuickTime 1.5 or so. Apparently the idea behind them is that all frames are actually tiles of a much larger picture that user can navigate without exhausting all RAM trying to decode it as one contiguous image. If you thought about ISO H.EIC (aka HEIF or AVIf depending on intraframe codec employed) you may be right, but also it got re-branded (and maybe enhanced a bit?) some time later as QT-VR (sometimes I think no matter how stupid modern multimedia idea is, it’s been implemented in QuickTime a couple decades ago).

Anyway, out of four such movies, one was recognised and converted by discmaster software, two were playable (with my player) after I hacked file type tag to be MooV instead of APPL, the last one could not be decoded at all because it was of an unknown type.

Luckily for me resource data of that movie contained the decoder in m68k binary format (sometimes I think no matter how stupid modern multimedia idea is, it’s been implemented in QuickTime a couple decades ago—or did I say that already?). And just by looking at the frame contents I knew it was worth REing as I could spot YUV codebook right at the beginning and it was definitely not Cinepak (or Compact Video as it was known back then). The name was “CDROM Video codec” with tag cdvc but that didn’t tell me much. The file was created in 1992 while Wickedpedia claims that SuperMac Compact Video was added to QT around that time as well.

Anyway, let’s move to the format details. The codec starts with 24-byte header followed by YUV or (theoretically) RGB24 codebook, the another 24-byte header (containing frame dimensions among other things) and finally data. Frame is split into 4×4 blocks and first there is an opcode sent containing block type and number of blocks (minus one) of that type. Blocks are known to have three types: 4 vectors per blocks, 1 vector per block (scaled 2x), or simply skip.

The concepts of codebook-based coding are the same as in Cinepak, even YUV conversion formula is almost the same (with simplified coefficients using multiplication/division by two only). The main difference is using just one codebook for everything and coding format—while Cinepak uses separate bit masks for block types, this codec uses opcodes (which is common for other fruity codecs). So this makes me wonder where this codec comes from and how it is related to Compact Video. Was it some kind of predecessor? Was it developed by Malus as a competition or based on the licensed technology? Why was it abandoned?

Even if I ended up with more questions, it was still a fun way to spend a Sunday weekend (the rest of Sunday was spent travelling to/from Lower Ulm and it’s a differently fun way to spend Sundays; but that’s not the point here). Who knows, with a new search approach I may be able to uncover a couple more of ancient codecs to look at.

P.S. Another fun codec for very early QuickTime was SIVQ (that’s how it was called, I’ve failed to find anything but the decoder for it). It was simply 128-entry 2×2 codebook (in RGB24 format) followed by codebook indices. Probably the name stands for “SImple Vector Quantisation”. That makes it the third proper VQ codec in QT (SMC is a slightly different beast; and RPZA is the first texture codec instead).

I could be writing about RedHat video encoder I just finished or work on REing DiVID1x on Paul’s request, but this was earlier in my queue.

Apparently on iVNC protocol there’s an option to use a custom iCodec for that. Since I was asked to look at it, here are my preliminary findings (more detailed bitstream description will follow eventually).

So packet starts with a byte telling payload type (0 – intra frame, 1 – inter frame, 2 – custom quantisation matrices for luma and chroma, 64 bytes each). After that the rest of data follows.

Intra frames code a series of tiles with tile metadata and actual tile content being separated into different parts. Frame data starts with two DCT quantisers followed by 24-bit big-endian metadata part size, then there’s metadata, and finally it’s tile data.

Tile metadata codes 3-bit tile type and the number of tiles having that type (0000–1110 mean 1-15 tiles, 11110 means next 8-bit value plus sixteen, 111110 means next 15-bit value plus sixteen, 111110 means next 22-bit value plus sixteen). Tiles can be of the following types:

• white tile—tile is completely filled white;
• last match—previous(?) tile is copied;
• upper match—tile above(?) is copied;
• black and white—one bit per pixel (0 – black, 1 – white);
• two-colour tile—almost the same but with two colours transmitted first (8-bit luma and 6-bit chroma values);
• DCT—tile data is coded with ProRes-like DCT;
• match tile—re-paint last recently used DCT tile;
• cached tile—re-paint DCT tile with 16-bit index from LRU cache.

Inter frames start with two bytes telling the number of coded chroma coefficients and the rest is single bitstream with 2-bit tile type and whatever tile content is stored. Tile types are: skip, DCT, match tile, and cached tile. The first type should be obvious, the rest is probably the same as in intra frames.

Also frame data is supposed to end with "mvs\0" but I guess this matters only for people trying to write a compatible encoder (or checking that the data was decoded correctly).

See, it’s a rather simple codec, so hopefully I’ll clarify some things (like cache behaviour, YUV coefficients and actual DCT bitstream format), document it at The Wiki and move to something else.

Since apparently discmaster.textfiles.com ran out of space (because storage is not cheap now thanks to “AI” companies), I have no new formats to be distracted with and have to resort to looking at the old ones.

As we all know, A**le single-handedly invented multimedia and the existence of IFF ANIM (that pre-dates it by a couple of years) should not confuse you. From what I could find, the earliest QuickTime samples can be found on Apple Reference & Presentation Library 8 CD-ROM that was intended for demonstration but not for the consumers. That disc actually contains three flavours of QT MOV: ordinary MOVs that can be decoded without problems, slightly older MOV that looks almost the same but has slightly different format (and features version -1 both in the header and in the metadata strings) and some extremely old files that do not correspond to the usual MOV structure. Those apparently come from the alpha version of QuickTime around year 1990 when it was still known by its code-name after USian artist of Ukrainian origin.

The first glaring difference is that atoms are not strictly nested but may contain some data beforehand. Essentially all atoms that normally have special header atoms have it before the rest. So instead of (pardon my S-expressions)

(moov (mvhd (trak tkhd (mdia mdhd (minf …))))

there is

(moov [moov header data] (trak [trak header data] (mdia [mdia header data] [all codec description data])))

Data structure is flatter too. In release QT version data for the individual tracks is grouped into chunks, so the header describes both the chunks and what tracks use what chunks (and what are frame sizes in the chunk). Here frames are stored as (offset, size) pairs.

From the accompanying decoder I can see it supported raw video, RLE and motion JPEG at the time—fancy vector quantisation codecs came later.

I hope to support it one day but there’s no hurry. Currently I’m more or less done improving NihAV tools and want to add some practical demuxers, muxers (and maybe even impractical encoders like for Indeo 4 or RedHat Ultimotion). And there’s na_eofdec still waiting to collect enough supported formats for its first release (tangential fun fact: recently I’ve added support for Electric Image format there which is mostly a simple RLE but its frames apparently have timestamps in floating-point format like 0.266 or 0.5).

P.S. I’ve also discovered MoviePak QT codec which looks like JPEG variant. Maybe one day when I’ll have nothing better to do I’ll look closer at it, but for now I have no desire to reverse engineer M68k code in unknown format.

Here’s a list of video codecs I found with the help of discmaster.textfiles.com (mostly by looking for .inf files with “vidc” in them). Maybe I’ll look at them one day, maybe somebody will beat me to it. In either case, it’s a nice reminder for myself.

• CSMX.dll—reportedly used for CSM0 codec, was bundled with some player, no known samples. Considering its small size it’s likely to be either raw YUV format or lossless codec;
• d3dgearcodec.dll—D3DGear lossless codec;
• elsaeqkx.dll—some Elsa quick codec;
• Esdll.dll—bundled with the same player, the strings inside it hint on it being an unholy mix of ZIP and MELP-based speech codec;
• ICS422.DRV—S422 or SuperMatch YUV 4:2:2 codec (I suspect it’s raw YUV);
• ICSMS0.DRV—SMS0 or SuperMatch VideoSpigot codec (from SuperMac, later bought by Radius);
• MCTCOD.DRV—MCTPlus Draw driver, supposedly offering 2:1 compression and has a bunch of FOURCCs registered to it: DRAW, MC16, MC24, MR16, MR24, MY16, MY24
• MyFlashZip0.ax—MFZ0 lossless codec. Looks like ordinary deflate wrapper really;
• NTCodec.dll—NewTek nt00 codec (I expect lossless or an intermediate codec). Looks like a simple fixed-fields packing;
• RGBACodec.dll—apparently Lightworks lossless RGBA codec. Simple RLE inside;
• Sx73p32.dll—Lucent Technologies SX7300P speech codec;
• TRICODC.DRV—Trident draw codec with a bunch of compressed RGB and YUV formats: rtc3, rtc5, rtc6, ty0n, ty2d, ty2n, ty2c, r0y1… Compressed in this context means packed using fixed-length bitfields;
• UCLZSS.DRV—Ulead LZSS codec aka uclz (obviously a simple lossless codec). As expected, it’s yuyv data compressed with LZSS;
• UCYUVC.DRV—Ulead compressed YUV 411 aka yuvc. Apparently it’s just 8/12/16-bpp YUV (fixed packing scheme, and 16-bit is simply YUYV/YUY2 and such);
• V422.DRV—Vitec Multimedia V422. Most likely simply raw YUV;
• V655.DRV—Vitec Multimedia V655 (I’ll venture a guess this means 6-bit luma and 5-bit chroma instead of improbable YUV 6:5:5 subsampling);
• VDCT.DRV—Vitec Multimedia VDCT.

Oh, and I’ve also found Eloquent elvid32.dll for EL02 FOURCC but it’s different since it has samples and it’s yet another H.263 rip-off.

That’s it for now, I’ll talk about FLAC video some other time.

Since I’ve done looking at game formats for a while (I’ve updated na_game_tool 0.5.0 release with the fixed extraction of old-style Cryo Interactive archives BTW), I decided to look at my backlog of unsupported (either mis-detected as audio-only or not detected at all) MOV files from discmaster instead. Of course the majority of them are files missing their resource fork (and majority of the rest are poorly-recognised data+resource forks in MacBinary II format; that reminds me I should improve support for it), and majority of the rest are files that I support already (like Duck Truemotion 1 or Eidos Escape codecs). And a good deal of the rest are Flash in MOV (not going to touch it). And yet there are a couple of files worth talking about.

• 911602.MOV—remarkable for having JPEG frame embedded in QuickDraw stream. It made me finally write a decoder for it (but without JPEG support, maybe one day…);
• a couple of .flm files turned out to have obfuscated moov atom (with something almost, but not quite, entirely unlike TEA if you care). Not something I’d like to support;
• La88110mN1Hz2_20-07.mov—it turned out to be raw YUV420 codec;
• Omni_324.mov—compressed apparently by MicroWavelet on NeXT. Intriguing but I doubt anything else can be found about that codec;
• Music Chase .tmv—a very weird little-endian MOV format with custom video codec (I think I mentioned it before but I haven’t progressed since);
• there was one Flip4Mac-produced sample (which name eludes me), but considering that it stores ASF packets inside MOV it’s not something anybody is eager to support;
• some undecodeable SVQ3 files—apparently they are encrypted;
• and finally there are some QT Sprite files, which seem to be (often deflate-compressed) frames containing several atoms with commands and occasionally image data as well. Sounds too hairy to support but again, maybe one day…

With AVI situation is somewhat better, there are some poorly formatted and encrypted/obfuscated files as well (plus damaged files from AOL archives that start at random position, so contents may be AVI file with some additional garbage in the beginning or missing few dozens kilobytes of initial data). Beside that I’ll probably implement PDQ2 decoder just for completeness sake. Eventually.

I said it before and I repeat it again: I like older formats for trying more ideas than modern ones. DMV was surprisingly full of unconventional approaches to video compression.

First of all, it uses fixed-size blocks with palette and image data occupying initial part of the block, and audio taking fixed size tail part of that block.

Then, there’s image compression itself: frame is split into 2×2 tiles that can be painted using 1-4 colours using pre-defined patterns, so only colours and pattern indices are transmitted. Those indices are further compressed with LZ77-like method that has only “emit literals” and “copy previous” (with source and destination allowed to overlap but with at least 4-byte gap for more efficient copying).

And even that is not all! I actually lied a bit because in reality it is 2x2x2 tiles since one pattern (using up to 8 colours) codes two 2×2 blocks in two consequent frames simultaneously. Now that’s an idea you don’t see every day. There are many codecs that code colours and patterns separately (like Smacker or System Shock video), some use a variation of LZ77 to compress data even further, but the only other codec I can name that coded two frames at once would be H.263+ with its PB-frames (and it was mostly interleaving data in order to avoid frame reordering, not sharing common coding data between frames).

Searching discmaster.textfiles.com for large unrecognised files returned more hits so I hope that some of the remaining formats will also feature some refreshingly crazy ideas. Now I only need time and mood for reverse engineering.

As I mentioned some time ago, Paul has shared a decoder for bytedanceVC2 codec. From the first glance it looked like H.266 rip-off. After a more thorough looking it seems to be more or less plain H.266.

There are some tell-tale signs of codecs using well-known technologies: simply listing strings in the binary gives you “VPS/SPS/PPS/APS malloc failed” (with APS being adaptation parameter set, which was not present in H.EVC but is present in H.266), “affine merge index is out of range” (again, affine motion was added in H.266), “lmcs_dec” (chroma-from-luma, another feature not picked up by H.265 but which was fine for H.266. Update: apparently it does something to the luma as well, still it’s H.266 thing), “alf_luma_num_filters_signalled_minus1” (a variable straight from the specification). I hope that this is convincing enough to claim that the codec is based on H.266 (or VVC).

Now what would a rip-off codec change? In my experience such codecs keep general structure but replace entropy coding (either completely or at least change codebooks) and DSP routines (e.g. for H.264 rip-offs it was common to replace plane intra prediction mode with something similar but not exactly the same; motion compensation routines are the other popular candidate).

I cannot say I got too deep into it but the overall decoding is rather straightforward to understand and from what I saw at least CABAC was left untouched: it’s exactly the same as in the standard and the model weights are also exactly the same (at least the initial part). Of course that does not preclude it having differences in DSP routines but I seriously doubt it being different from the standard (or some draft of it).

And with that I conclude my looking at it. Those with more motivation and actual content to decode are welcome to try decoding it, I have neither.