Kostya's Boring Codec World

Since my previous question got a satisfying answer (even two) maybe I’ll get some answers for this question. It’s not been bugging me but it’s been with me for a long time.

First, the background. What we have:

• there’s a significant demand for video editing software for all those content creators on various video hosting platforms;
• there are some commercial products that got mentioned rather often (usually with complaints about their stability);
• there are some freeware or shareware products (less mentioned but you can see them in paid advertisements) that are built on opensource technologies (many of those have libavcodec in some form, one even turned MEncoder into .dll form; I checked the programs specifically to see if they have some interesting decoders bundled with, no luck so far);
• there are some open-source video editors as well like Cinelerra, Kdenlive or LiVES that keep existing in obscurity (AviSynth or Blender are more popular but they’re a different class of programs);
• most of the building blocks for input/output formats support or filters are already there (and if not I’m pretty sure Paul B. Mahol can add more on request);
• there’s a certain traffic cone shaped foundation (maybe VLC secretly stands for cône de Lübeck video) that is known for making an opensource video player that majority of people have heard of and use—on various platforms too;
• beside the player (which can also stream and do other things) they have supported other projects (like x264 or dav1d).

So the question (or rather a tree of related questions) is: why can’t those people organise and make a popular open-source video editor? Are they not interested in a product with less then ten million potential users? Are there any principal issues in creating a video editor that I’ve not seen (I remember VLME which did not even look like a serious attempt)? Or maybe there are some ideological reasons preventing its creation?

To me it looked like it’s possible to build an opensource video editor, it should use mostly the same blocks as video player, it should have some demand and it is possible to make a popular open-source product. So why it is not done, what I’m missing in the picture?

…and silence will be the answer.

Here’s another thing that has been bothering me for some time. As we all know RDBMSes more complex than SQLite have data management that is more complex than simply writing all data into one file sequentially plus you have database users belonging to different roles with different permissions on different databases. And you’re supposed to connect it via the usual networking mechanism even if you’re running it on the same machine. And advanced features include multi-machine interaction like sharding, replication and such.

So the question is: if this all looks like a standalone operating system then why enterprise-grade RDBMS are running as applications instead of their own OSes build on some kernel to manage CPU cores, storage and network interface? And in the age of containers we don’t really need to have an app version even for legacy version, just replace the driver (or merely its settings).

But maybe like the last time I wondered why GPUs do not have raytracing, I’ll learn that it’s a trend already and I’m just oblivious to it.

When trying to look inside an old game you might have a problem if the executable is 32-bit one running with some DOS extender.

Most people do not know (and those who know probably want to forget it) that beside the usual 16-bit COMs and EXEs (that can have several segments and sometimes an overlay too—an external file for loading code or data from disk) you can have 32-bit DOS applications with a very special interface to access so called protected mode. Essentially you have two executables bundled together: actual 32-bit executable in its own format and the loader for it (or a stub to call an external loader usually called dos4gw.exe). Kinda like PE format but more flexible.

So what’s the problem with loading them? For starters Ghidra does not recognize the extended format and loads just the loader (an external LX loader plugin did not help either; but when I tried to load some ActiveX components for XVD decoding, their format was detected as LX). Another problem is that some extenders (like the one from CauseWay) also compress the payload. Luckily enough there’s DOS/32A extended that is shipped with a utility to extract the so-called linear executable which you can’t run but you can study it.

And this is not hard but somewhat tedious since (at least in my cases) it can be loaded only as raw image. From the LX header you can extract the position of actual segment data, segment sizes and their base addresses; then you can use that information to create custom memory segments for them. Now you have some (usually two) segments and one problem: the segments do not address the full virtual address but rather an offset e.g. if you have code segment starting at 0x10000 and data segment starting at 0x90000 you might have mov eax, [0x4bc] which really should be mov eax, [0x904bc]. That is why the first hundred kilobytes or more of LX is dedicated to so-called “fixups” that tell you how you should correct the addresses at the specific places in loaded data (or code). Luckily Ghidra allows patching the addresses in the instructions and re-assigning segment base addresses to have less places to patch (and calls/jumps almost always use relative offsets so it’s not an issue).

And how to find those wrong addresses for specific data item to patch? Memory search for low word part of the address works just fine.

Overall, maybe it is not the most convenient way to deal with this format but it works sufficiently good for me. Who knows, maybe in the future I’ll look at more games for not yet covered formats. At least now I know how to do that.