I could’ve written this post much earlier if not for the pressing need to upgrade OS on my main machine. It took time and it will take even more time to configure it to a more useable state; but I’ve made enough rants about software updates already so back to the original topic.
I’ve finally reverse engineered CDV format. While the format itself is quite remarkable, the fact that I had to deal with CauseWay DOS extender packed executable is more important as it offers possibilities to explore even more games. So I’ll document how to deal with it here so I can refer to it later (or maybe somebody else will find it useful but that’s highly unlikely).
So (in case you forgot) CauseWay DOS extender is a special loader allowing to run 32-bit code in DOS environment. Some loaders are standardised around Linear Executable format and can be freely interchanged, Phar Lap P3 format apparently was used on FM TOWNS as well, but this one uses a non-standard format and packed to boot.
There are actually two compressed formats associated with it, both starting with “CWC” signature, window bits for the LZ compression that follows and 32-bit unpacked and packed sizes. The main difference is that one of them is used for compressing user data and employs 16-bit flags packing and another is used by the loader itself to pack code and employs 32-bit flags packing. Good news is that it got open-sources at some point and you can refer to the original assembly code for writing an unpacker (which I did). There’s even a handy utility called CWC.EXE which can compress and decompress user data format which may help you validate the results.
An executable in this format contains several compressed segments, one right after another. First segment contains the compressed code for the extender itself, others seem to contain some data (and I could not properly unpack them), the last segment will contain actual executable code. Just locate it (in my experience first segment always starts at offset 0x4B0 and you can follow them from there), unpack and load as raw 32-bit x86 code. It’s relocatable so addresses start from zero by default. That’s it!
Now about CDV format. It’s been used in some games by Synthetic Dimensions PLC (along with CWC-compressed image data) and its video compression leaves an impression of FLI mixed with PNG. There is an old frame compression format where each line starts with a compression method (0 – no changes, 1 – RLE, 2 – skips and updates, 3 – skips and RLE-compressed updates and so on) and there’s a new frame compression format which employs LZ77-based compression. But unlike many other formats, in CDV most compression schemes can be viewed on high-level as skip and updates lengths with update blocks being coded using several opcodes. To elaborate a bit, traditionally RLE with skips would be coded as “skip N1 pixels, run of M1 pixels, skip N2 pixels, copy M2 pixels, …” with skip codes either being mandatory before single run/copy opcode or each of three possibilities occurring arbitrarily. Here e.g. method 3 codes data as “skip N pixels, decode 8 compressed pixels (run of 4, copy 1, run of 3)”. Similarly in CDV LZ compression you have “skip or decode N pixels” on the higher level and if pixels are compressed, it can be a mix of literals, LZ77 match pairs, or runs (signalled by offset being zero). Of course it’s not something groundbreaking but I like this kind of small deviations from the mainstream implementations.
That’s it for now. I intend to re-visit ART format and figure out its algorithms but no promises on actually accomplishing that.