H.264 decoder postmortem

August 27th, 2023

I mentioned before couple of times that NihAV has its own functioning H.264 decoder. And after my failed attempts to use hardware accelerated decoding instead, I spend some time trying to optimise it but eventually gave up. On one hand it’s fast enough for my needs, on the other hand it’s too tedious to optimise it further (even if I can spare time on it, I’d rather not).

To put it into perspective, initially it was about three times slower than libavcodec one without SIMD optimisations, now it’s only about two times slower (with SIMD turned on it’s about five times as slow, feel free to laugh at me). But at the same time playing 720p content (and I have next to no files with larger resolution) in multi-threading mode takes 20-25% of the core so it’s not that bad.

So how the cycles are wasted and is there a potential for serious optimisation?
Read the rest of this entry »

A moral dilemma

August 22nd, 2023

Disclaimer: the question presented in this post does not affect me in any way but it’s still perplexing enough to make it public. Also please note I don’t mention names (neither of people nor software) as this post is not about shaming them.

As any other area of human activity, multimedia has its own share of, ahem, eccentric people that are obsessed a bit too much about their project. For example, there’s a guy who constantly informs the world about even the slightest advancements of his image codec with his own unique image quality metric. Theoretically it should be interesting but the author keeps ignoring the useful advice he gets (like making his code work with a different image size or explaining how his metric is better than the others) in hope that somebody else will get interested enough to fix that for him, as his lack of free time prevents him from doing anything but minor improvements to the code. Maybe it’s not the wisest approach but it does not harm anybody so good luck to him, maybe one day there will be a breakthrough and it will get at least a limited fame. But there’s another example that came to my attention recently, which is significantly more disturbing.

So there’s a certain codec that has niche popularity for its speed and decent compression ratio. Since it was proprietary and somewhat popular, a certain person (not me) has reverse-engineered it and added decoding support for the format to FFmpeg. The reaction from the creator was rather baffling at the time, it was like he felt the control of that codec was wrestled from him. Oh well, enough time has passed with no other issues arising. But last week the same person who REd the decoder announced about that he’s working on an opensource encoder for it, and that’s when the situation exploded. The format creator in a tone that I think is called passive-aggressive told that it’s essentially a stolen work and that it made him stop on a new version of the work. And what is significantly worse and greatly disturbing, his words sound like he got a depression from it or even suicidal thoughts. Even while I have reasons to believe that the encoder in question is going to be an original work (i.e. not a plagiarism; REing format to ensure compatibility is also permitted by the law in many countries) the possible consequences are still deeply disturbing, to say the least.

Thus several questions arise: what should be the best course of action to resolve situations like this one? Was an opensource implementation even for a decoder a mistake and should it be removed entirely? Should the author better communicate his wishes that there should be no alternative implementations whatsoever in the first place and should the others honour it if the product becomes too popular for its own good? Even if the law permits it, what about the morals?

I can only be happy for the fact that I’m not involved in it at all. In either case it would be nice to know the answers—and even nicer is they will never ever be useful.

P.S. Corporations are not people so do not try to project the situation to them. And if they feel offended their lawyers will tell you that (so far I think only N*llym*ser tried it).

P.P.S. If you think I also suffer from similar psychological issues—maybe you’re right, I’m not be eligible to judge. At least I do not try to force my stuff onto others, I don’t even post anything at public places except in this very easy to ignore blog.

The original CfL codec

August 17th, 2023

As most of you don’t know and don’t care, modern advanced video codecs may use the special prediction mode called “chroma from luma” where, as it’s obvious from the name, the chroma components are reconstructed from the luma using some coefficients. And what do you know, I’ve found a codec that used this approach back in 1997.

So there’s a French company called Kalisto Entertainment and back in the day it developed a codec for the cutscenes in some of its games (at least Dark Earth and Nightmare Creatures). 15-bit RGB video is split into three components and each is coded separately using simple LZ77-like method (i.e. it’s either RLE mode, or copy with an offset from the current or previous frame). The twist comes from the fact that those components are split into tiles (usually 20×20 ones) and each tile has coding mode and two sets of scale/offset coefficients, so for each tile one of RGB components is selected as the base one and two others are coded as the differences from the scaled (and offset) base value.

So one component plane contains the base components for each tile (which may be different for each tile) and the other two contain the differences for the predicted non-base components (which, at least in theory, should be mostly zeroes and thus better compressible). So when some people wonder if it’s time for video codecs to perform optimal component decorrelation on per-frame basis, here’s a practical codec from the last century that did it per-tile.

On good russians

August 7th, 2023

As I mentioned in one of my previous posts on this topic, I consider russians to be rather a viral mindset than a nation: they don’t have clearly defined territory (as they consider they whole world to be it), they have no own culture (it’s either stolen or for-display set pieces that get no relation or acceptance with russians; well, you can argue that their widespread prison subculture is their own but how does that make things any better?), they lack human qualities like compassion required in this age (i.e. a millennium or two ago russians would be no different from many other nations but the times have changed), and the worst of all—they try to convert everything they come in contact with into russia (either by conquering the territories and committing cultural and actual genocide or by demanding that the other countries do everything their way because it’s too hard for russians to learn other country language and customs). And yet there are naïve people believing that there are “good” russians even that’s an oxymoron. Usually that comes from a belief that if russian say they’re against war and current government that implies they’re against other things as well. Here I’ll try to stratify russians by their empathy and activity:

  • plankton—like the namesake those russians have almost no will of their own and are merely flowing with the currents. If you ask them about their position, it will be more “for all good and against all bad” and they never take interest in politics. They’re always supporting the government but when it changes they’ll support the new government equally half-heartedly (just remember what happened in Rostov during the laughable coup attempt). The sad thing is that they’ll readily resort to violence if they’re permitted by the authorities. They have no compassion (there’s enough evidence how they’ll cheer to the war crimes their army commits and to the spectacle of the same war criminals dying as long as it’s a good show). Slave-owners and dictators may call them good but since I’m neither I can’t;
  • moths—those are almost the same as the previous category but they can have their own opinion not fully in line with the mainstream one and even—gasp—be against the government. The only problem is that they don’t act on their words, saying that they’re mere moths and can’t do anything. Of course they’ve never tried to find it out if they can actually do anything or not. For those I have only mild despise because they don’t deserve a strong emotion;
  • sell-outs—not originally russians but who became ones usually by being seduced with russian money. You might’ve heard about russian actors like Steven Seagal or Gérard Depardieu. There’s nothing wrong with coming to another country in pursuit of better work opportunities. But there’s a difference between selling your skills and selling your dignity and calling such people good is like calling a native advertisement a good article;
  • chameleons—those russians actually have a position, the problem is that it changes depending on circumstances and who asks about it. To give a concrete example, in the first days of February 24th some propagandists from russian television said to their friends that they wear their half-swastika symbols but mean it as a hidden support for Zelensky. Obviously they may voice support for anything but in reality they’re concerned only about their own well-being. Somebody not familiar with that feature may call them good, not knowing that the words they heard were empty, I call them disgusting;
  • white coats—that is a semi-official term for those who put themselves over the others like in the famous scene from Monty Python and the Holy Grail: —Must be a king. —Why? —He hasn’t got shit all over him (I vaguely remember a russian joke about somebody first spraying shit over others and then appearing in a clean white coat, that must be the origin). Such people pose themselves as morally superior to others, impartial judges and so on—forgetting they have no ground for that or trying to proactively evade the possible blame. As with the previous category, somebody naïve enough to believe their words without checking their background may believe they’re good but they should know better;
  • armchair Napoleons—those do not even try to hide their ambitions. In the “worst” case they want russian army to be better equipped and fighting (forgetting that russian traditions like stealing and corruption prevent it), in the “best” case they want russia to conquer all of the world. Or if it all fails they’re fine with russia nuking the rest of the world. You need to be a psychopath to call them good (again, search elsewhere);
  • and finally russians with human faces. Those pretend to be actual humans and often serve as an example of “good russians” in the West. In reality though sooner or later they show their real russian face. They might be against the current government but what are they going to do? There’s a post by some random russian that sums it up the best: “This regime will fall, Navalny will become a president and restore the country. And then we’ll get back at you, Ukrainians!” You can dismiss it as being just a single deviant voice, but in reality prominent figures from russian “opposition” demonstrated the same behaviour and chauvinism as the officials (like spreading the false claims about blown up Kakhovka HPP or not understanding why not everybody would like to interact with russians in general). To me it seems that they try to maintain the usual russian imperialism by keeping up ties with other countries (so when russia has more strength it can come there “in order to protect oppressed russian-speaking citizens”). So they’re about as good as telemarketers are your friends.

If you think this does not apply to somebody specific, try to get the honest answers for the following questions: are you against the current war? are you against the war just because it inconveniences your life (with sanctions and possible partial mobilisation)? do you think only the russian government is responsible for starting the war? do you think that russia should be held responsible for the war crimes it committed (e.g. paying reparations)? do you think that Crimea belongs to russia? do you think that russia should withdraw its forces from all occupied areas (Abkhazia, Armenia, Belarus, Ossetia, Syria) as well? do you think that something substantial should be done about russia in order not to make this scenario repeat again? do you consider the idea of dissolution of russia in order to make national republics acceptable? do you understand that russian writers often followed imperialistic agenda and thus other nations have reasons to ban their works? and finally, do you agree that russians are not superior to other nations? Hint: not all of those yes-no questions have “yes” as the right answer so you need to think before answering them.

russians usually give themselves away by starting to cry that Crimea is russian, always has been and giving it to Ukraine was a historical mistake as the existence of Ukraine itself. On the other hand, if somebody passes the test perfectly then probably you’re not dealing with a russian at all.

Meanwhile the only real good russians are mentioned in reports like this one:

NihAV: giving up on hardware acceleration

August 3rd, 2023

After having several attempts on trying to add hardware-accelerated decoding support for NihAV I’m giving up, the reason being the sorry state of it in general.

I’m aware of two major APIs for hardware-accelerated video decoding for Linux, those are VDPAU and VA-API. Plus there are some specific toolkits e.g. from Intel but from what I remember those are even more complicated.

So, VDPAU has only bare-bone documentation without actual explanation what is expected for each codec in order to decode it. VA-API turned out to be even worse: it points out to 01.org for documentation which no longer exists (and redirects to some Intel’s page blurbing how great they are at open source). And web.archive.org shows that that page essentially contained a link to libva and libva-utils repositories plus some references to the projects that have VA-API support implemented. “…so shut up and go away” was not written but implied.

At least VA-API has three crates implementing its bindings in Rust and not just one not updated in four years like VDPAU but how usable are those? There’s FeV that seems to support JPEG decoding only (and has a strict warning against AMD GPUs), there’s libva-sys that is a pile of auto-generated bindings and there’s cros-libva. The latter seems to be the cleanest one and most actively developed (too actively developed to my taste as it changes base APIs every couple of months). Unfortunately it’s still not exactly clear how to use it for H.264 decoding (and the cros-codecs crate provides equally confusing API). And the final straw is that it seems to be intended for single-thread use only by design, which means it’s not possible to use with my library (e.g. my player uses separate threads for audio and video decoding, so I can’t use the standard decoder interface for hardware-accelerated decoding without some ugly hacks).

Oh well, I’ll work on improving my own H.264 decoder performance—while it’s not much fun either at least it’s clear what I can do with it and how it can be done.

P.S. This reminds me of the situation with ALSA. From what I heard it’s one of the worst documented subsystems in Linux with too flexible interface, to the point that it took help from ALSA developers to make at least MPlayer support ALSA output. The most probable reason is that it’s common to smoke weed in Czechia (where ALSA was developed), but what is the excuse for the other libraries?

Why I work on NihAV

July 30th, 2023

I started NihAV as a more or less toy project to play with different concepts and try new stuff like finding out how vector quantisation works or attempting to write an encoder. Having enough experience with libavcodec and libavformat, I did not want to touch them again (and still don’t) and there was a hope that rust-av will provide a viable albeit limited alternative for multimedia playback (it still hasn’t). In theory I’ve achieved my original goals—NihAV supports decoding a lot of exotic formats (some of which are not handled by any other open-source project), it even has some encoders and its own transcoder tool and there’s even two players (one for audio files, another one can also play videos). So I could relax and do something else entirely but yet I’m working on adding new features to NihAV that take a lot of effort and do not bring me joy. Why?

Read the rest of this entry »

NihAV: updated for Rust 1.69

July 27th, 2023

Since I had nothing better to do I decided to optimise my H.264 decoder a bit more, and that required a rather recent version of rustc that supports sym construct in asm!{} (so I can reference data tables in the inline assembly). Why this specific version though? I picked whatever was both recent enough to support the aforementioned feature (and older version had multiple micro version releases which hints on some problems with them) and not too recent either (again, I’m no beta tester of the compiler and I don’t need other shiny features).

And while at it I decided to make the code a bit more up to date. cargo-clippy is still annoying with its default warning about all-caps names and some lints that changed names and their suppressors no longer work. Getting rid of some leftover hints for the old versions of the compiler (like explicit drop()s for the objects borrowing code and some type hints) was nice though. Inline assembly is still only halfway done, especially considering that using const in it won’t be possible in stable for a long time and sym sucks compared to GCC inline assembly (it provides just a symbol name and you should magically know for yourself how the target platform works in order to make it possible to load it correctly; on AMD64 it’s rather simple but on aarch64 and on 32-bit ARMs that depends on target OS and PIC mode). Who would’ve thought that assembly may be platform-dependent! Looks like the current solution to that problem is to expose current configuration to the user so it’s up to you to check all environment variables and write the appropriate code. And of course even that solution will be available some time in the future since the developers haven’t thought about it at all.

Anyway, now my H.264 decoder features some more assembly optimisations and decodes video even faster than before. Though I fear it still takes too much CPU for the comfortable playback of my typical content so I’ll have to dabble in the hardware video acceleration. NihAV is a learning project after all.

Simple things

July 20th, 2023

Simple things are usually the hardest to accept and follow. Here I’ll list the things considering the current situation and we’ll see how those will be understood by the world:

  1. There’s currently a world war going on. The definition of world war concerns mostly not the number of countries involved but rather that the goals of war (or its consequences) affect the world order in general. In this case if russia wins it means that the old system built on respecting country sovereignty and resolving international conflicts in peaceful manner via Useless Nations (formerly United Nations) does not work and any country with the nukes can do whatever it likes. And when the temporarily existing despicable mistake known as russia will lose the war, this may lead to its dissolution as well as making entities like UN and NATO reform or perish. And it’ll impact the future of China too;
  2. russia has demonstrated that it is a terrorist state countless times (trying to disrupt the world order by force is the very definition of international terrorism), but in addition to that it demonstrates that it’s not above the economic blackmailing as well. Just look at the recent development of the grain deal—it did not merely stop participating in it until its simple demand of fulfilling its countless demands is met but also started missile strikes at Ukrainian ports (again) and threatening to start a war with the countries that will keep participating in the grain deal without russia. And of course spewing obvious lies instead of saying directly that it’s racketeering;
  3. People who commit such crimes are either arrested and isolated in prisons or executed, so they can bring no harm to the society. Armed people (especially if they’re shooting during the arrest) are often shot on spot to eliminate the immediate danger (that’s not the best outcome but it’s an acceptable one). Countries should have the same treatment, out of self-preservation if not anything else (and stop point at nukes, russia demonstrated that it poses more nuclear threat when nothing is done about it);
  4. Speaking of isolation, it should be maintained airtight instead of trying to earn money while hoping that whatever russia does with your resources won’t be used against you later. I’m not so sure about the business risks of (usually French) companies that still have their russian subsidiaries operating as usual but if they suffer from the reputational losses in Europe and their businesses would be confiscated in russia, that would be a completely foreseen outcome. Also considering the current isolation and slow implosion of russian economy, it’s hard to tell what good the income earned there can bring (as you can’t transfer those money from russia and there’s risk of losing them entirely);
  5. When NATO talks about eliminating corruption as one of the demands for the candidates, it should serve an example and do something about the glaring example of Hungary. EU should take note as well.

Again, those are very simple things to understand but apparently not for the countries or large businesses. For now though, I find it ironic that I could travel with less restrictions and was significantly less ashamed of my country (and even its government) when I had Ukrainian citizenship than now when I’m a German citizen.

TM1 encoder: probably done

July 19th, 2023

After some trial I decided to release what I’ve done and probably not return to it ever again.

Currently my encoder can encode 15-bit TrueMotion 1 format with different block sizes. It’s probably not very adjustable but there’s not that much to adjust really. I’ll talk why I gave up on 24-bit mode (again!) below, for the other options here’s a condensed version: it does not matter. I’ve tried encoding files with an alternative delta set and it resulted in significantly worse picture quality (but at least encoded frames were usually larger as well); as I mentioned in the previous post, only the first codebook makes sense for 15-bit data (as other two codebooks waste space on coding delta value 7 which is not used in 15-bit mode). Inter mode uses simple skip block as I didn’t bother to think about the possible threshold but it works good enough anyway. In theory I could calculate gradients to determine what sub-block sizes to use for each frame (as I did in Indeo 3 encoder) but again, I decided not to bother.

Now, here are the reasons why 24-bit mode is much harder. For 15-bit mode you can easily calculate deltas for each (decorrelated) component independently rather easily—and the coding method allows selecting deltas in fine-grained way too. In 24-bit mode you have chroma delta pair that updates red and blue components and luma delta pair that updates red component with one value and green and blue components together with another value. In theory decorrelating just green and blue components should help but there we hit another issue: the amount of possible deltas is good enough to represent different delta values occurring during the prediction stage. Essentially you can’t process each component independently and should rather apply deltas as 32-bit values to the 32-bit pixel value, then unpack it and see that the individual components aren’t far enough from the desired ones. It is not that hard to implement but it essentially means writing a second TrueMotion 1 encoder that processes 24-bit data in an entirely different way. Considering its limited use and the fact that it shrinks down horizontal resolution in two times—the coduck (that’s their very original name for it) always processes blocks of two 32-bit words but now those are two 24-bit pixels instead of four 15-bit ones. In either case, even if I see how it should be solved I’m not going to actually do it.

I need to find myself a better task to undertake.

Restarting the work on TM1 encoder

July 15th, 2023

Back in February I wrote about my failed attempt to write TrueMotion 1 encoder. And since I was bored and really had nothing better to do, I tried my hoof at it again.

Last time it was 24-bit encoding, now I tried to approach 15-bit encoding instead and got some results. I guess the moral of the story is that you should not overthink it and use the simplest approach to coding.
Read the rest of this entry »