Archive for the ‘NihAV’ Category

Random NihAV news

Thursday, July 24th, 2025

Since I have not tweaked any weights and have not made any releases, I’ll just write about some stuff I’ve been working on but have not released yet. Meanwhile librempeg got support for a bunch new formats too so its changelog may be a more interesting read. Anyway, this post is about what I have (and haven’t) done.

First of all, I’ve finally fixed an annoying problem with VA-API decoding on one of my laptops. Counterintuitively, it turned out to be faster to request hardware to convert native surface into some other format (NV12 into YUV420) and then use it instead. This made decoder CPU usage drop under 10% at last. Probably it can be optimised further to reduce load on graphics subsystem but I’d rather not mess with OpenGL unless it’s really really really needed.

Then I expended support for two formats in na_game_tool. VDX (used in The 7th Guest) had a different format version for the game demo. It still employs two-colour VQ but data for intra frames is split into separate parts for masks and colours, and inter frames code updates to masks and/or colours for each block instead of independent decoding. Additionally thanks to Discmaster I’ve located DPEG version 2 which employs completely different algorithm from the version 3 (painting 4×4/2×2/1×1 squares for intra and skip/update for inter).

I’ve also discovered some new interesting formats like Lantern MOV (which codes DIB changes using 16-bit RLE and there’s a probably related older version in IFF instead of RIFF). I’m considering making a sister project to na_game_tool to decode various formats like this one, formats coming from Amiga, recording formats and such—for all the formats that I’d like to try decoding but don’t want in main NihAV. I’ll write about it when I actually have something to write about (i.e. when I have a name and enough formats for 0.1.0 release). Another curious find was fractal video codec—not the ClearVideo but something with fourcc FVF1 from Images Incorporated. Who knows, it may be interesting to RE.

And finally here’s what I really wasted too much time on: Motion Pixels decoders. It has rather annoying binary specification (like using segment registers to address decoder context variables) that decompilers refuse to translate and from I heard it’s impossible to run on anything newer than Windows 95 or NT4. Nevertheless the formats pose some interest.

From what I saw long time ago, MVI2 is MVI1 with certain extensions, and MVI1 is surprisingly close in the structure to MVI in its own format files—and Gregory Montoir has reverse engineered it long time ago.

So I started by reimplementing that MVI decoder (since I can debug its behaviour against known working implementation) while trying to understand what it does. I got it more or less working (reconstruction is still not perfect but at least it’s recognizable) and my decoder supports other files (found with Discmaster of course) that trigger demuxer bugs or have different subsampling modes.

Then I moved to implementing MVI1 decoder applying the differences found in the binary specification. While it still does not handle decoding properly (both the pictures are garbled and I don’t use all deltas stored in the frame), at least it proves I’m on the right way. Hopefully it’ll decode properly soon and then I can add MVI2 features. Of course it’s a useless format nobody cares about, but apparently I do.

PMM support for na_game_tool

Friday, June 27th, 2025

While I suffer from thermal throttling, I don’t have much progress to report. Mostly I did some quality of life fixes for NihAV. For example, BaidUTube decided to switch AAC streams to HE-AACv2 in some cases, so I spent some time debugging and fixing SBR decoder in order to get rid of annoying chirps.

But finally I decided to be more systematic and tried to compile a list of known FMV (and FMV-ish) games and what formats they use. In the end I got a list of about four hundred known games, though of course I did not look at all of them: some games are not easy to find, others are console games which I’d rather not touch. SEGA formats are more conventional (often it’s just Cinepak or TrueMotion in AVI), while 3DO or CD-i is something too exotic and for Playstation you have better luck with librempeg.

Anyway, while compiling this list I discovered that PMM format is used in three games, so I decided to implement it at last. It is supposed to contain rather simple quadtree-based video coding with 15-bit pixels that may be updated per-component, and audio may be coded with one of several simple compression ways. Unfortunately format description in The Wiki was somewhat lacking and did not describe audio type 3, so I had to look at the binary specification.

Audio compression type 3 turned out to be real DPCM (while type 2 is not DPCM but rather 16-to-8 bit mapping scheme a la A-/μ-law). It codes packets of 16 bytes where the first byte is delta shift and the rest are 4-bit signed delta values.

The format itself turned out to be more curious than expected as it may be in either endianness (which affects not merely chunk sizes but header fields and code bits of the video data as well). And to push it further, it apparently also code frames as raw palettised data (using SEGA Saturn palette format).

But that’s not all, Area 51 game actually uses YUV colourspace for some of its videos so sometimes you get BGR555 and sometimes it’s YUV555—with YUV to RGB conversion coefficients sometimes transmitted in a special chunk. I suppose it improves compression ratio a bit.

In conclusion I say the usual thing: looking at those old formats is usually more interesting than looking at the new ones. Back in the day developers had fewer opportunities to borrow ideas from the standard formats (except for FLI of course) and threw different crazy ideas in order to see what sticks. For instance, some French codecs liked to perform motion compensation tied to affine transform (i.e. rotating and/or mirroring that block); some codecs used RLE operating on pairs of bytes or copying backwards; some codecs stopped at coding block with two colours, while others used several or even put those colour combinations into a frame-level codebook. Even those two codecs (Reaper and Xilam DERF) that would look like ordinary JPEG clones used an interesting mix of bytes and nibbles to code coefficients without wasting too much space or resorting to Huffman coding. And nowadays people are lazily waiting for H.267 (and if anybody would use H.266 for real) and AV2 (which should’ve been released about four years ago). No wonder people have little interest in multimedia.

P.S. It’s not likely that I’ll work on supporting another game format soon. Even disregarding the heat wave, I have other things I’d rather work on (and maybe there will be something to report eventually).

NihAV: now with TealMovie support

Wednesday, June 11th, 2025

Back in the day I looked at the format and recently, to distract myself from game formats, I decided that it might be a good not the worst idea to implement decoding it.

And in the course of doing that I discovered some things that make it even more peculiar. For starters, it flips every second sample in its ADPCM coding. I don’t know if it improves compression in this particular case or it was done just to be different. Similarly split sub- or sub-sub-blocks are coded in コ-order instead of more traditional zigzag order.

But there are more interesting things about it. For starters, the file is organised into blocks instead of frames. First block always contains metadata (streams parameters, title, creator and such), next blocks contain one or more video frames (which you have to decode one after another; I implemented frame parsing for finding out frame boundaries but that’s inelegant solution), and last blocks are used to store audio. This means demuxer either has to demux audio frames after all video frames are sent or jump places in order to maintain synchronisation. Since this is not na_game_tool, I picked the former. The samples are short, so it’s easier to decode them to AVI+WAV and remux properly (or decode both streams to AVI and make AVI demuxer handle unsynchronised streams better—but that’s a task for another day).

Another surprising thing is that there is 16-bit RGB support, done in a very peculiar way. Frame decoding remains the same, except that now frame data is actually a pseudo-YUV frame with two chroma planes following the luma plane. And of course the conversion is done using one of two tables (depending on file version) using the formula yuv2rgbtab[(u + v) * 128 + y]. I guess it’s coding luma, colour difference and colour difference difference here.

And finally, intra frames in TealMovie are stored raw. But when frame width exceeds 160, it is stored half-size.

That’s why I’m looking at those old formats: there’s significantly more variety there in employed coding methods and storage format nuances. Not all of them make much sense but sometimes they’re entertaining or so original that it makes you wonder why such approaches got no further development.

P.S. Maybe I should take another look at the handheld console video formats.

P.P.S. But I think I’ll have to do some boring things instead. With BaidUTube changing its available formats it seems I finally need my own MP4 muxer. In either case that’s easier than to fix libav.

na_game_tool 0.4.0 release

Tuesday, June 3rd, 2025

At last a new version, now with over a hundred various formats being supported!

About a third of them are console formats of mostly raw variety (you need just to figure out how tile data is represented) but some of them required some reverse engineering as well (e.g. Road Avenger employs LZ77-based compression). Another third comes from known sources (either The Wiki or ScummVM source code. Last third is from my reverse-engineering efforts on DOS games.

What’s next? I don’t make plans, but I like the idea (suggested by Paul of librempeg fame) to make the tool more versatile. So version 0.5.0 should at least support extracting data from game archives: some of the currently supported video formats can be found only inside large game archives, so why not provide a way to obtain the files as well?

Beside that, I’ll probably focus on more complex codecs, namely the ones that are based on other codecs like Cinepak, JPEG or Smacker. I’m not sure I’ll be able to find another dozen formats to RE but at least there should be something new beside merely archive extraction.

Meanwhile grab the current version here (or don’t, it shan’t make a difference).

na_game_tool: yet another bunch of formats

Sunday, May 11th, 2025

So, what’s been added since previous report?

  • Road Avenger BIN—as mentioned in an update to the previous post, done;
  • Reaper FMV (from Rollcage game)—decoding works fine but the reconstruction does not because of imperfect IDCT implementation on my side (and the fact they don’t use any kind of clipping to make sure the result will stay in range all while employing different tricks like converting double to int by adding a large constant, storing the result as double and read the second half of it as an integer). Overall, I decided not to count it and threw in a couple of easily REd formats described in the next entry;
  • a pair of other BIN formats from Data East—they’re the same as Road Avenger BIN but not even compressed;
  • Ecco the Dolphin LMS—another raw-tile Sega CD console format except that tile data is arranged more curiously there (128×128 frame is split into 128×32 strips consisting of 8×8 tiles that are arranged in columns). Also it seems to use implicit grey-scale palette (unless there are some external palettes hidden elsewhere), at least watching those documentaries in this way is fine;
  • Interplay M95—it’s been described on The Wiki since 2006 (not by me of course) but nobody bothered to implement a decoder for it until now;
  • Psygnosis SMV (from WipEout game)—I looked at it back in the day but of course it’s been on The Wiki since 2012. Codec 2 was not described completely right there (partition codebooks are stored in the end, after block indices and not before; and it was not immediately clear that block palettes are stored in global frame order and not in partition coding order) but nothing that a bit of debugging can’t fix.

In either case I’ve filled self-imposed quota of a dozen of originally REd formats and started to pick other formats to support (three done, about nine to go). I have noted about three formats worth supporting and if I can’t find more good candidates—well, maybe there are some more low-hanging Sega CD formats…

And as usual, for more supported console formats, both simpler and much more complex, there’s always librempeg (and that’s not talking about other things it offers like the countless audio and video filters).

P.S. Since I don’t want to do a separate rant, I’ll mention some tangentially related things here.

First, a search for more SMV samples on discmaster returned no unknown real SMV samples but some PMV samples with .smv extension including one that could not be decoded (because of zero root chunk length—I’ve fixed that) and a couple that had nothing wrong with them but were not detected as Zork PMV at all for some reason.

Second, I actually used discmaster to check if there are some interesting ARMovie files. Sadly it still lack an option for omitting supported files so I searched for acornReplayAudio files and filtered out the sources containing actual ARMovie files without video track (somehow I don’t want to check all of ~6000 files so if one or two files coming from same CD are audio-only I excluded it from the further search). After all this winnowing I’ve found two samples with Eidos Escape 102 codec. It’s simple and I’ve REd it some time ago so finally I’ve managed to write a decoder for it. At least I don’t have to search for the samples by the magic string like I did back in the day… Sadly there’s no good way to detect old TCA format so I’m not even going to attempt looking for it.

na_game_tool: another bunch of formats

Wednesday, April 30th, 2025

Short summary for Paul: nothing interesting. I write it as a report of done work and to provide some clues for somebody looking for ancient formats.

After I did EGG format support, I remembered that samples.mplayerhq.hu had a directory dedicated to that and some other game formats (from Smacker to Bink 2), so I decided to look at it and see if I can support some of those. Spoiler alert: there were low-hanging fruits there.

What got supported:

  • ICOM video—this one is used in their Sherlock Holmes Consulting Detective series. Initial frame is unpacked, the rest are coded as sequences of skip and masked updates. I’ve managed to RE it from the sample files only, it’s that simple!
  • Klondike Moon VID—also raw first frame plus simple RLE for the following frames, and I’ve also managed to RE it from the samples;
  • Maelstrom ANM—this one uses RLE for both intra and inter frames;
  • Manic Karts AMF—simple RLE with a special large value skip or run modes that encode skip/run length in the way known as Xiph lacing;
  • The Amazing Spider-Man vs The Kingpin BIN (Sega Saturn game)—again, I’ve REd it from the two known sample files despite this being a console game. I had some vague ideas about how the console works and indeed it proved to be raw data for tiles/palette/scroll planes (even if it was send as a partial update in some cases) that I simply had to assemble correctly. Just reading some console specifications of its video system was enough.

What I looked at but haven’t REd (yet?):

  • Knowledge Adventure Movie—I looked at it some time ago and it’s not a format I’m eager to revisit;
  • Road Avenger BIN—this format seems to compress tile data with some custom LZ77-based compressor with a signature “sarc20“. I’ve managed to locate the bit in the binary specification responsible for the unpacking but no references to what and how it’s used. Nevertheless I hope to finish REing it with my new knowledge Update: done. It’s simply one plane with tiles coded and palette transmitted before the compressed data;
  • Rollcage Ripper FMV—I REd it long time ago, I just need to locate my notes and finish it. Update: it’s in progress, decoding works, reconstruction is far from good though;
  • Rocketz VQM—I failed to find the code responsible for the decoding (and Ghidra decompiler refuses to work on many functions there), and DosBox debugger reports that these files are read in 16kB chunks, so no luck on guessing their internal structure. Maybe one day…

In either case that makes it ten original formats added since version 0.3 and about ninety in total. Two more originally REd formats and another dozen of formats REd and documented by others (those are easier to come by) and my goal for the next release is complete.

One game, two rare-letter formats

Thursday, April 24th, 2025

As I’m still in a search for formats to implement in na_game_tool for 0.4.0 release, one game brought two formats at one—and both start with a letter I had only one or two formats before.

I’m talking about ID4 or Independence Day. Its Sega Saturn port uses EGG format which has not been REd before. But first I was able to locate PC version of the game with its animations in YQS format, so I started with it.

It turned out to be a very curious format, with YUV frame coded per plane using quadtree and two separate streams for pixel values and for control bits, eight streams in total (there’s IMA ADPCM audio as well and fixed-size vector codebook for luma). Essentially plane is split into 8×8 (luma) or 4×4 (chroma) blocks which may be skipped entirely (if the control bit is not set), filled with single value obtained from the stream (again, if the next control bit is not set) or split into four sub-blocks with the similar approach. 2×2 chroma blocks may be skipped, filled or coded raw; 2×2 luma blocks additionally may use input pixel value as a codebook index and use that 2×2 vector (flipped or not, depending on control bits).

After that I actually looked at EGG. It had similar structure but with some changes: frames are always 20kB and do not contain audio, values are big-endian and bitstream format for 2×2 luma blocks differs. I still had to resort to the binary specification for it though—loaded CPE file in Ghidra as raw SH-2 binary, located animation names and started attempts to disassemble data around them. Eventually I got the function that looked very similar to the one in PC version (except that it decoded just luma instead of doing all three components and converting macroblock to RGB). Thus I got a working decoder for this format as well.

That’s how I got two more formats supported, which makes it almost 42% of my goal. The problem is still mostly to locate the formats that can be supported (most of the ones I haven’t implemented need a lot of work to unpack the binary specification). In either case I’m in no hurry and it was nice to learn more about old formats.

NihAV: even further encoder improvements

Wednesday, April 9th, 2025

Since the last time I wrote about it somebody actually decided to test my transcoder on a bunch of fringe formats. The main issue there was wrong audio output (since I didn’t bother to check the actual input format, it often passed e.g. floating-point audio as 16-bit PCM, resulting in complete garbage). Additionally I’ve fixed a stupid bug in Indeo IVF demuxer (so now both known samples are decoded fine) as well as improving support for some other corner cases.

And there was one fun failure only my transcoder could have: when video streams starts with skip frame(s) and ZMBV encoder was selected, it refused to encode. I’ve added a work-around so now it simply presumes that frame to be black and goes forth normally (if you encode into raw video, such initial frames would be skipped instead).

So whatever can be decoded seems to be decoded and re-encoded just fine (but there are probably more cases waiting for their turn to be discovered). And there are some more formats waiting for an open-source decoder (in case of Motion Pixels the wait is likely to be very long).

na_game_tool: new season start

Sunday, April 6th, 2025

It’s been some time since na_game_tool release and I’ve finally started working on adding new formats for it again (for v0.4 the goal is to have a hundred of supported formats including a dozen of fresh ones, currently it has only about eighty). And since newly added formats are somewhat remarkable, I decided to talk about them.

Spidy Ani from Imperium Galactica. The name sounded vaguely familiar and indeed I’ve encountered it before. I did not bother to support old sprite format since they are just animations for which you need to know the background image and palette. Version 2 contains both palette and (optionally) audio, making its support much more reasonable. As for the compression, it actually keeps the old RLE method and introduces a new one (where low values mean copy length instead of a literal) plus it adds optional LZSS compression. The best part is that it’s not been reverse engineered by somebody else but it has been documented (in open-ig wiki) as well.

Shadowcaster cutscenes. This is a format used by a game published by Origin, it turned out to be RLE optionally further compressed by LZSS (I shan’t blame you if you see some similarities here). The main difference is that the format contains commands for e.g. playing an external XMI track (or stop playing it). And instead of one palette it can store several and switch between them arbitrarily.

Castles 2 (CD version) greyscale video. This one is very special: the game has colour cutscenes in Interplay MVE format (usually with castles) but beside that it apparently has some greyscale animations lifted from public domain films. The files have .m extension and start with “BYON PIZZA” string, definitely not hinting at Byon Garrabrant, who was the lead programmer of that game.

It was extremely hard to read the binary specification for it as the executable stored that code and data in executable overlay. That means that it was in a special part of executable that gets loaded on demand, thus lacking proper references to the data and even functions. Luckily it turned out that I didn’t need it at all.

One of the sequences turned out to be just one black (or white) frame. Seeing that the frame was exactly 1/4th of raw image and filled with 0x40, I decided to calculate a histogram of some other (larger) frame. It turned out that indeed if you output values in 0x00-0x3F range as is and repeat 0x40-0x7F values four times you get exactly full image. Then by looking at the reconstructed image I understood that it codes data in 2×2 blocks (and in inverse mode so 0x00 means white and 0x3F means black). After that figuring out that codes 0x80-0xFF in subsequent frames are used to signal skips was trivial.

That was fun. Hopefully I’ll encounter more interesting formats in the future.

P.S. I’ve also added PI-Video decoding support to NihAV. The codec was used at least in one multimedia encyclopaedia and did some non-conventional things with LZW coding. I’m still not sure that it was a good idea because codec developer may be russian but since I could not find any information about him (or the companies involved beside them being Western European), I’m giving it the benefit of the doubt.

NihAV: encoder improvements

Friday, March 21st, 2025

Since I feel I’ve done enough (for a moment) on na_game_tool, I decided to work on the rather (always) neglected NihAV. And specifically on the nihav-encoder tool.

As you can guess from the name, it is a tool for re-encoding input file(s) into something else (or something the same). Since I don’t have much need in transcoding anything, I use it occasionally to test encoders (especially for collecting information by running encoding with different parameters on the selected sample set) or a muxer. Essentially if video is encoded correctly that’s enough, audio de-sync or even drop-outs (because not all encoded packets were actually muxed) is another thing. Finally I’ve decided to improve the situation and make it more usable.
(more…)