Kostya's Boring Codec World

As usual, I don’t have to say what has been put to words in much better way by somebody else already.

At the banquet, last winter, of that organization which calls itself the Ends of the Earth Club, the chairman, a retired regular army officer of high grade, proclaimed in a loud voice, and with fervency,

“We are of the Anglo-Saxon race, and when the Anglo-Saxon wants a thing he just takes it.”

That utterance was applauded to the echo. There were perhaps seventy-five civilians present and twenty-five military and naval men. It took those people nearly two minutes to work off their stormy admiration of that great sentiment; and meanwhile the inspired prophet who had discharged it—from his liver, or his intestines, or his esophagus, or wherever he had bred it—stood there glowing and beaming and smiling, and issuing rays of happiness from every pore—rays that were so intense that they were visible, and made him look like the old-time picture in the almanac of the man who stands discharging signs of the zodiac in every direction, and so absorbed in happiness, so steeped in happiness, that he smiles and smiles, and has plainly forgotten that he is painfully and dangerously ruptured and exposed amidships, and needs sewing up right away.

The soldier man’s great utterance, interpreted by the expression which he put into it, meant, in plain English—

“The English and the Americans are thieves, highwaymen, pirates, and we are proud to be of the combination.”

[…]

The initial welcome of that strange sentiment was not an unwary betrayal, to be repented of upon reflection; and this was shown by the fact that whenever, during the rest of the evening, a speaker found that he was becoming uninteresting and wearisome, he only needed to inject that great Anglo-Saxon moral into the midst of his platitudes to start up that glad storm again. After all, it was only the human race on exhibition. It has always been a peculiarity of the human race that it keeps two sets of morals in stock—the private and real, and the public and artificial.

And here’s a link to the full writing in case you haven’t read it already.

Today on “things that Kostya cannot change so he rants about them instead” is something different from the usual political or open-source political rants.

There are several groups of people whose occupation is (in theory) to evaluate certain things. So (again, in theory) you can call them the metric for those things. In practice though they rather do the opposite and try to make things conform to the valuations they give, or at least to make public perceive those things in the way that would confirm the original claims (and truth be damned!).

Of course some would see nothing wrong with it, others would even try to tell you that they’re always right because they cannot be wrong and thus only their opinion is the true one. Well, I’ll present three examples and see for yourself.

Let’s start with the most prominent example, namely lawyers. In an ideal world, lawyers are a part of judicial process, making sure that the side they support is represented fairly—and that means that judging is done according to the laws, without glaring mistakes or prejudices. In practice though lawyers tend to get associated with paid justice, meaning that quite often the outcome of the trial or litigation depends on the pay-grades of the lawyers involved instead of actual known facts (or even laws). Which sometimes leads to the fun systems as the British one with two mandatory kinds of lawyers (barristers and solicitors) and USian one—resembling quantum dynamics—where you can call lawyers elementary particles responsible for any interactions between entities (except that quantum dynamics is easier to comprehend).

Then there’s another often disliked group of people called philosophers. In theory philosophy is a way to explain the world or some of its aspects. So one would expect philosopher to be a thinker who studies the world (or part of it) and makes some conclusions about how it works and what implications that gives for the rest of the things. For instance, science may study human morals as a thing emerging in collectives and affecting interactions between members of those collectives, while philosophy may ponder how morality defines human itself and what should be considered the ideal moral. But the modern philosophers seem to work in the reverse: first they start with a conviction (quite often a small one and benefiting them directly) and work up from that to build a system that will provide an excuse for their beliefs. Of course this is unlikely a modern trend, but history preserved enough examples of real philosophers for any epoch and different countries as well—which is hard to say about the modern world.

And finally art critics. One would naïvely expect them to be people with certain tastes who appraise certain kinds of art (paintings, sculptures, books, movies, video games etc etc) and tell public their opinion about it. You may like them or not, agree with them or not, but in either case such reviews should not merely give an abstract score but also provide an explanation of what was done right, what could be improved, and what hidden qualities may make it even better than your first impression told you. There’s a reason why people still remember and quote Roger Ebert (of Chicago Sun-Times) or Scorpia (of Computer Gaming World). But the majority of the modern art critics seem to start from the premise of having to praise the reviewed product (often, apparently, out of fear for the salary and other benefits—disappointed owners of the badly-reviewed product may stop advertising in your media or provide early access to the next products they release and so on) and construct up the review leading to the goal without mentioning the actual reasons. What makes it worse is that often it’s accompanied not by the notion “if you love X and Y then this is definitely a thing for you, otherwise you may want to skip it” but rather “if you don’t like it you’re a dumb bad person”. Thanks, I still remember a bit of Soviet Union to reject it at a visceral level.

So there you have it. Of course this effect is nothing new but I felt that for some reason I need to say it, so here it is.

Sometimes I remember that Matroska exists and today I also remembered how it came to existence. Its author proudly admits mixing all the buzzwords of Web 2.0 era like XML, Semantic Web etc etc and coming up with that format. Since we’re in Web 3.0 era we should have something more modern.

That is why I’m calling for a modern multimedia container format to supplant the outdated formats of the old. It should encompass all the features that make the modern Web great:

• binary JSON as the backbone of the format;
• central repository for the downloadable descriptions of the parts of the format (but not codecs themselves! Think of it as of MXF specification if it helps);
• blockchain (as well as clusterchain and framechain);
• metaverse integration;
• decentralised storage (so that the container may refer to some data in the cloud as well as on the local disk; even MOV could do something like this);
• and of course AI!

Some of you may ask where AI can be applied in this scenario. The answer is obvious—transforming input data for better compression (let alone generating metadata or enabling better integration with other Web 3.0 products). A good model should be able to do the same savings like Matroska did by e.g. shaving off common headers bytes from each frame but without special mapping. An excellent model may generate the content by the embedded description instead of transmitting AV2 video. And of course the central repository will contain the description of models and parameters to be used (in addition to the descriptions of better representation of container parts layout). The possibilities are limitless!

Proposals should be sent to the Alliance for Open Media , I have worse things to deal with.

I’ve been wanting to write this post for a long time, with a focus on the difference between hobby project and product and about NihAV only. But a recent FFdrama made me re-think both the structure and the conclusions.

Apparently there’s another surge of developer’s discontent in jbmpeg for having mushroom treatment (not for the first time and probably not for the last one). IMO they need to realize the project is as free and democratic as Soviet Union and you need simply to agree to the things proposed by the General Secretary (definitely not the leader)—that would save time and nerves of everybody involved. As I wrote countless times before, I do not fear for the future of that project as it can keep such existence indefinitely long and here I’ll try to present my reasons why.

First of all, revolution a la libav won’t work—Michael has learned the lesson and he won’t be kicked out again (not that it really worked in 2011 but now there are no chances for that).

Second, if you split out and form an alternative, it has not so many chances to replace it. And if you decide to write anything from scratch your chances are next to zero. The rest of this post is dedicated to answering why.

Recently I re-read The Mythical Man-Month which tells not only about the author’s experience designing RedHat OS/360 but also presents more general observations and ideas. And right in the beginning he talks about the difference between a program, a programming product, and a programming systems product. Essentially, a program something a programmer writes and that it works for him on his system; a programming product is a program with documentation and support; and a programming system product is that works as a component in a larger system. And moving from one stage to another requires an effort several times larger than the previous one (I’m simplifying a lot and probably misremember something—so you’d better read the original book, it’s a worthy reading anyway).

Here we have a similar situation: writing a tool just to do things for you is straightforward, even I have managed to do it with NihAV; making it into a product requires offering much wider support for different platform configurations (for example, my videoplayer has VA-API hardware decoding enabled by default while it’s not available, say, on Windows and you need to switch that feature off there in order to build it) and different features (e.g. nihav-encoder works for testing encoding per se, but lacks ability to encode input into a good intermediate format supported by other players and encoders). And it gets even worse if you try to make it into a library ready to be used by others—beside the usual things like documentation you’re expected to guarantee some API stability and a certain level of quality. So while I may not care that my app panics/crashes in certain circumstances, it’s significantly less forgivable for a library. And of course achieving such quality level requires a lot of unexciting work on small details. Debugging is even worse.

Suppose you decided to create a fork and work from that. Then you still have a much worse position—you may have the same codebase but there are no killer features you can offer and you don’t have recognition. libav managed to succeed for a while since it was supported by some distribution maintainers—and even then users complained because the de facto brand name was replaced with some unknown thing. And I guesstimate 40% of current jbmpeg developers contribute to it in order to upstream the changes they make while using it in their employer’s product or pipeline. So how can you convince those companies to use your fork instead? And that’s not taking patent situation into account which makes a substantial support from any large company for your project rather improbable.

Good thing I’ve never intended NihAV to be a competition, but what about other projects? rust-av died because of lack of interest (Luca claims that he started it mostly to learn Rust and see how performant it can get—mission accomplished, no further development required). librempeg fares better but I doubt that Paul wants to deal with all the demands that other parties make for the honour of your stuff being included into their distribution (or being used without even a credit).

Another thing that needs to mentioned is that multimedia is no longer an attractive field. Back when I started to dabble in the field, it was rather exciting: there were many different formats around—in active use as well, and people wanted to play them not only with the proprietary players. There were libraries and players supporting only a specific subset of different formats, like avifile or libquicktime or DVD-only player. Nowadays it’s usually a combination of H.26x+AAC in MP4 or VP9/AV1+Opus in WebMKV, all formats having specifications (unless you lack Swiss Francs to pay for the ones from ISO) and new formats are not introduced that often either. Of course, we might have H.267 standardised soon but who uses even H.266? When was the last time you heard AV2 development news? The codec was supposed to be released a couple of years ago, did I miss it along with AV3? Do you remember Ghost audio codec from Xiph? Of course Fraunhofer will keep extending AAC patent lifetime by inventing new formats and naming them like OMGWTFBBQ-AAC but who really cares?

That is why I believe that no matter how dysfunctional jbmpeg is, it will keep existing in this undead state indefinitely long as it’s good enough for the most users and there’s no compelling reason (e.g. new popular formats or radically different ways to process data) to switch to anything else. The only winning move is not to play.

Paul of librempeg fame informs me about his achievements occasionally (and in my turn I try to remind the world time from time that this project exists and may provide you functionality hardly found elsewhere, like various filters or console formats support). And his recent work was implementing inflate routine specifically for the multimedia needs. This made me think if it makes sense to have a custom implementation for deflate decompressor and packer in multimedia project when zlib exists and I think it makes perfect sense. Of course in NihAV I NIHed it because the project concept demands it and it was a nice exercise, but in more serious projects it makes sense too and below I’ll try to explain the reasons.

Zeroth of all, a quick reminder about flexibility. RFC 1951 merely specifies the format, so the implementations can output varying bitstreams that will correctly decompress the data. Back in the day when I worked on it I mentioned how you can compress data in different ways, dedicating more time to achieve better compression. And there are more tricks not mentioned there like parallel compression.

Now, the first reason why to have your own implementation is that you can adapt it to your custom needs. As Paul demonstrated, inflating data into a custom format might be beneficial. If all you need in 99% cases is unpacking data from one buffer into another or to a frame buffer (which has padding so you have to output, say, 27 bytes, then skip 5 bytes, output 27 bytes and so on) you can do without all additional functions and not bother with a sequence of calls to partially inflate data.

Then, there’s a question of consistency. You cannot be sure that a new version of zlib will produce the same output as its previous version (I vaguely remember that there was a small scandal when GitHub releases were re-generated using different deflate implementation and resulted in a lot of headache because of old archive hashes being no longer valid; or there’s a story of some Linux distros replacing zlib with zlib-ng resulting in failed tests; and even “no compression” format may change apparently). The case with liblzma is probably a good demonstration of other reasons why it’s not always wise to rely on third-party components.

And finally, you can not merely adapt interface to your needs, you can tune it to handle your data better too. There’s a reason why there exist compressors targeting e.g. genome data. So when you compress image data, it may be beneficial to search for matches around the position right above the current line first, and the presets for the compression levels may be tuned to the different sets of trade-offs. After all, deflate is often used in screen capture codecs where real-time performance is more important than the compression ratio. But who can imagine people tinkering with the encoder trying to improve its performance in a multimedia project?

I hope this helped to convince you that there are circumstances where NIHing something may prove worthy. As a rule of thumb, if it’s easier to implement yourself than to re-use an existing library then maybe you should do so. That it the right way, and the other way lies left-pad.