Archive for October, 2022

Visiting multimedia grave

Monday, October 31st, 2022

When people ask why I call the search division of Alphabet Inc Baidu, I answer that I do it in spite to muddle their search index and mostly because they remind me of a Chinese totalitarian company. And recent news only reaffirm such views.

As you should remember, Baidu is famous for its graveyard for the killed projects—it even has a separate alley for the messenger apps. And looks like it prepares a plot under a concrete duck for burying some multimedia formats (which makes it interesting to me).

The history of multimedia formats at Baidu essentially started with the purchase of On2 and releasing VP8 in WebMKV format. Then VP8 was mostly buried since VP9 was created (some of it remains hidden inside WebP format), VP9’s turn is near since VP10 is here to succeed it (under the name of AV1).

In the recent news though it turns out that Chrome is deprecating its support for JPEG XL, a format developed mostly at Baidu and the only one properly standardised. But as we all know, Chrome currently controls the Web and removing support for it means that the format will remain obscure. Kinda like in a Soviet joke where a foreign tourist asks in a shop why there’s no caviar and hears that there’s no demand for it—and as he observed for a whole day nobody asked for it indeed (in case it’s not obvious people in the USSR didn’t ask for caviar at the shops because they knew it would not be sold there; see also Baidu Stadia).

And because it was not enough, people spotted that WebP2 has changed its status to experimental, meaning that it won’t be supported either.

So we have, VP9 buried in favour of AV1, JPEG XL being buried in favour of AV1F, WebP2 being buried in favour of AV1F (which is AV1 still frames in MP4) and the original WebP is likely to follow the suit. Now consider that AV1 is recommended to be distributed inside MP4 instead of WebMKV and you’ll fear about the future of that container as well.

I guess now all is left for them to do is to adopt Baidu Lyra as non-experimental codec to purge Vorbis and Opus not created by them and then bury it in favour of AV1-based audio compression. That would make a nice collective grave of formats killed by Baidu to make space for AV1.

So, do you know when AV2 should arrive?

A final look at SIFF

Saturday, October 29th, 2022

This is a resource format (as one could guess from its name) used by Beam Software for various resources in its games. The first time I had to deal with it was when some anonymous contributor contacted me with a REd decoder for videos in Lost Vikings 2. I adapted for FFmpeg (wow, it’s been fifteen years ago) and added sound support as well.

Recently I decided to revisit it (previously I tried to adapt decoder for 15-bit video but could not make it exactly right for some reason). So I looked at it, it turned out to be exactly the same algorithm but depending on whether palette is present in the frame (yes, 256-colour palette with RGB555 entries) pixel data is read either as byte palette indices or raw 16-bit values. I also looked at The Dame was Loaded game and its FCP format turned out to use exactly the same compression but with a different frame format (16-bit sizes, different audio/palette/video order and so on). And the game audio uses packed 12-bit PCM (two samples without low nibble are stored in three bytes) which is not something you often see.

And with that I’m done with game formats for a long time. Probably I should write a toy encoder or two now.

Looking at Talisman animations

Thursday, October 27th, 2022

While the situation in Ukraine is stable bad so I keep looking at game formats instead of playing those games instead.

Apparently there’s a German adventure game from mid-1990s called Talisman and that’s about all I can tell about it. Yet it seems to have rather interesting animation format that’s worth talking about.

It turned out to be codebook-based compression with a codebook shared by a group of frames (typically 20-30) and actual codebook being created from a transmitted codebook data mixed with the static codebook (an interesting fact: unlike many other game formats, this one not merely allows seeking inside, there’s a special chunk preceding group of frames with the same codebook that contains an offset of the next such chunk). Tile operations are either copying 2×2 tile from another part of the frame (optionally updating one pixel in it) or reading it raw.

While it’s not the most complex format ever it turned out to be interesting enough. And with that I’ve probably exhausted the game video formats to look at. I’ll probably revisit SIFF but that’s about it.

A quick look at PDQ2

Friday, October 21st, 2022

So there’s this new service discmaster.textfiles.com for searching for files from some old disks and discs at archive.org and out of curiosity I took a look at what it offers.

While it’s physically impossible to check all hundreds of millions of files it indexes (even just videos), I took a look at one rather curious category called aviAudio. There are about six hundred files of such type found so I decided to check them to see what kind of beast that is.

Some files turned out to exactly AVI audio files—AVI files with single stream that turns out to be audio (I don’t know why but this happens). The rest of the files were AVIs with unrecognized video track (just what I was hoping for!). One set of the samples come from some game with IPMA codec, other sources mostly feature Motion Pixels video (which is horrible both to RE and to support so I’d rather not touch it), one sample was special “codec” for some russian erotic game (which means complete garbage even by russian game standards) that is really just obfuscated Indeo 5. And finally there was a single PDQ2 sample (from some Incredible Hulk game demo from 1997).

An interesting thing is that it’s a DOS game with built-in AVI demuxing and decoding of this codec. The codec turned out to be a lot like various lossless codecs (so its FOURCC probably stands for something like “pretty damn quick codec, version 2”). It splits video into 4×4 tiles and emits up to three streams stored one after another: tile opcode (copy previous tile, copy previous data with an arbitrary offset, skip tile, raw tile), variable offsets and tile pixels. Additionally frame data may be packed further with LZ77-based scheme. As I said, a rather familiar scheme for lossless video codecs and it makes me wonder whether it’s a codec developed specifically for the game(s) or it was borrowed from somewhere else.

Another bunch of interesting game codecs

Monday, October 17th, 2022

While Iran wages a proxy war with Ukraine (and I can easily believe they’re not against doing it to the last russian), I’m still trying to distract myself by looking at various video formats in games I have no intention to play.

Finding games with unknown formats gets harder and harder (or at least more tedious). So here are two games with an interesting approach to video compression.

Dark Earth packs frames as separate planes with 5-bit components and each plane is compressed using a peculiar modification of LZ77 scheme: depending on a bit set, it either codes a run or a forward reference in either the same or previous buffer. Additionally it codes it using non-bytealigned stream (run mode is coded in 13 bits and copy mode takes 25 bits).

There’s also Defcon 5 game (the one from the mid-1990s, do not confuse it with a game with the same name but from a previous decade). This one employs a different method I don’t remember seeing anywhere else: painting squares. Of course you’re used to various codecs splitting tiles in blocks and filling them (from older codecs using quadtrees to H.26x and its rip-offs). This one does it differently though: there is a fixed set of tile sizes selected for each video (I’ve seen only 32, 16, 8, 4, 2 set though) and frame data is essentially a list of commands telling at which offsets to paint a tile of this size with a given colour (those offsets may be completely arbitrary). It may be not the most efficient compression algorithm but it’s quite interesting. Also the container specifies palette as fixed and changing colours—first part is stored in the header and the second part is transmitted with each frame. Again, it’s not something extremely effective but I can’t easily remember any other format doing it that way.

I’ll probably try to do something different next time but I consider time spent on looking at these formats well-wasted.

Looking at yet another game format

Saturday, October 15th, 2022

Since I have nothing better to do and I don’t feel a desire to write a lengthy rant about what’s wrong with this world (beside the fact that russia still exists of course), I keep looking at various games I can find in order to see if they have some interesting format. Last time I’ve managed to locate an elusive Indeo IVF sample, now it’s rather curious FMV format.

So there’s an unpublished PC game called Highlander: The Last of the MacLeods which is apparently some kind of 3D action game based on the animated series (one of those, the franchise seems to have a lot in it beside the one good movie). And it turns out that it has a three dozen of FMV clips in proprietary format (stored inside mov.dat but extracting them is trivial).

From technical point of view the format is rather simple but video compression is rather peculiar: each frame in coded independently by painting each 4×4 block with 1-16 pixels using a certain pattern. That’s common but this data is packed with a simple history-based method: depending on a flag a value is rather read raw or reused from a history buffer depending on two previously decoded values. And now there’s a twist—this compression is applied twice.

Overall, it was fun to look at the format even if I don’t care about the game itself and the video quality is rather poor (probably thanks to the rather low bitrate and fixed palette all those FMVs share despite half of them being digitised animation clips and half of them being recorded in-game 3D animation). Still, now I can play the files with NihAV if I ever get such desire.

Looking at true IVF

Thursday, October 13th, 2022

So while russia demonstrates once again what a hysterical führer and his similarly minded generals would do, I’m trying to do something to distract myself from the thoughts about the losses Ukraine suffered in last couple of days (and previous couple hundreds of days too).

Accidentally I’ve managed to find a sample in IVF format—the original streaming format for Indeo codecs. Despite the expectations it was not a renamed AVI file which made it even more interesting. So I took Ghidra and binary specification and started implementing it.

The format by itself is rather simple and the only interesting thing is that it transmits video data in passes: first it’s just bare minimum keyframes and all other frames as drop frames, then some inter frames are sent, then more data is sent and more. Unfortunately there’s no obvious field to tell where each part starts so I simply assemble frames in memory (which is not effective but still better than the original creating temporary files for assembling fragments).

Then there was a problem with decoding them: my existing decoder expected a frame with sequential structure while in this case fragments are transmitted out of order. Normally you’d get all four bands for luma plane and then a band for each chroma planes but in this case it may be one or two bands for luma plane, then chroma plane bands, then some zero padding and then the rest of the bands for luma (if they were transmitted at all). In the reference implementation the demuxer seems to parse and re-assemble the frame while I ended up writing a slightly different decoder for this scalable mode. It seems to work fine so now NihAV support for Indeo family of formats is even more complete than ever.

P.S. The Wiki had only the entry for Duck test samples format (that has the same extension) so somebody had to correct that. Which also makes me think about things like Duck AVC, AVS being an FMV game format or Chinese non-Duck AVC rip-off, IEEE AAC and so on. This confusion deserves to be written about but don’t expect me to do that.

Model of man and social/economic systems

Thursday, October 6th, 2022

Here’s some random thoughts about how some subjects should’ve been explained at school to make understanding it easier. Or maybe they were and I was unlucky.
(more…)