Kostya's Boring Codec World

Yesterday that Derek’s talk at Demuxed got to me for about the fourth time and I was asked about my opinion on it as well. I can take the hint (eventually), so here’s what I think.

Unlike Derek I’m a major nobody with some interest on how the codecs are working, to the point that I’m not afraid to look at their binary specification and sometimes even implement a decoder. Anyway, I’ll try to give a short summary of the points he presents and what I think about it.

First he talks about HEVC adoption being features and not bandwidth saving. I can believe that, especially if there are companies like Nvidia with its notoriously bad H.264 decoding support (which on certain models could not decode videos of certain sizes let alone more advanced features) and not going to update it for the new features. And with integer-based transforms you want higher bitdepth for better coefficient representation: I heard from x264 people that the scene used 10-bit H.264 to compress 8-bit content exactly because for quality considerations. Similarly my colleague said more than once that he watches Netflix videos in 4K downscaled for FullHD to hide the compression artefacts.

Then he mentions VPx series of codecs and that it did not enjoy that much of adoption in the wild. I would probably mention VP10 (aka AV1) there already but other than that there’s nothing to add here.

After that he moves to list who (and why) are not eager to adopt new codecs. Again, nothing much to add here.

He concludes the first part of the talk with the potential prospects of AV1 and VVC. I’d rant more about how patent extortionists spoil the whole landscape and how AV1 is lacking on support even from the members of AOM but he had a time limit.

The second part of his talk starts with a tepid take of H.264 becoming a JPEG of video codecs (implying that it’s become old but well-used familiar codecs with some extensions everybody prefers not to think about). Probably this is true.

He goes on with mentioning MPEG-2, MPEG-4 ASP and MPEG-4 AVC codecs and that they were implementable and reasonably documented. I’d say that the only unreasonable thing about the documentation was ISO pricing of the standards (and still is).

Then he laments on the lack of the knowledge preserved from the older generations. I share his emotions but some of his examples are not good. Last time I tried to search for some MPEG document, it turned out that some of them are open to MPEG members only. Other stuff was quite well documented in e.g. textbooks on analogue television but who reads them nowadays. And another stuff is obscure because it’s taken for granted. For example, we know that the Earth is spherical (in the second approximation) and even Ancient Greeks knew that—but do you know how they got to that conclusion? Can you tell without resorting to the web search anything beside shadows at noon in various places? And do you really need that knowledge in order to exploit the Earth shape? Another point is that some decisions are made for non-technical reasons: I heard that MPEG Audio Layer III was made such an ugly beast partly because a certain company had a patent on QMF used in Layer I and II so it forced it into Layer III design as well. But of course such things won’t get into textbooks.

This is followed by a cursory remark on two things: the same stuff repeatedly proposed for newer codecs. I still believe that codec development is hindered by the computing resources available (e.g. the celebrated chroma from luma prediction mode in AV1 started as a rejected proposal for H.265—but now it’s acceptable). Back in the day there was a joke that a perfect encoder is the one producing MD5 hash of a frame and letting the decoder to find a suitable data for it. With AI-based approach this might be no longer a joke (have you ever heard about Stable Diffusion?).

And with this he comes to the logical conclusion that out of the sheer complexity of the field it is stagnating: nobody understands the whole picture, the stuff is not taught well so hardly anybody can get into it (and they have to learn it by trial and error), and some of the stuff gets reinvented again and again. I mostly agree with that but I should point out that some codec authors still write books about their stuff (back at VDD15 I actually showed one of those VP9 guys the chapter with description of AVS2 from such book and he managed to find a lot of similarity between those two codecs). But while codec overviews are rather common, books on writing a good encoders for them are virtually non-existent. It’s exactly the same situation as with the compilers: back in the day you were supposed to know how to write a parser, a lexer and perform some optimisations—nowadays it’s “take LLVM, implement new syntax and let its huge swarm of optimisation rules that nobody comprehends in full do the work for you”.

Finally he thinks on what could be the possible ways to deal with it (beside some vector instructions for easier operations in the codecs none of it looks feasible to him—or me). And he ends with an uplifting possible outcome of codecs becoming the next COBOL—not going anywhere but remaining largely irrelevant. It might go this way indeed.

---

As a bonus I’d like to present my own thoughts on this topic. Back in the day I ranted about H.264 reinventing a wheel so the things were going downhill back then already.

I guess there are two processes at work here: codecs by themselves do not bring that much money as before (partly because people expect stuff for free, partly because you can’t vendor lock them so you have to interoperate and compete with other parties). And when codec licensing started to bring serious money (probably starting with MP3 and H.264) there were not only research institutions willing to fund R&D but a lot of patent trolls got active who’d readily exploit a patent from an old bankrupt company to get their share of the royalties. That’s how we got several H.265 licensing pools and Opus patent pool (from entities not related to the codec development in any way). This stifles new codec adoption with patent concerns and their development by (equally reasonable) concerns of not getting enough money for that research. I’d refer you to Leonardo Chiariglione’s blog for a more detailed picture but those who are aware of it have read it already and others have no desire to do that anyway.

As for the education part, I guess there’s not that much interest to fund its development either. I remember reading some texts from Dmitry Vatolin, the head of MSU Graphics and Media Lab (known for their quality measuring tools and regular codec comparisons). There he points out that they have the problem with preparing students: there’s a lack of specialists so students get offers before they have a chance of graduating, which leads to the lack of interest in training such students, so the lack of specialists increases… Probably it’s not as drastic in the West and companies there understand the need for independent research entities and sponsored university labs (and not only as the source of people; and not see it as a constant competition threat) but I don’t know much. He also points out at “digital dementia” as one of the problems with the modern students—and this problem is more universal, so you probably should not expect any improvement with codec development coming from the academia.

And there’s probably a ceiling on the media format specifications which is not worth exceeding. For audio it’s about 56 kHz at 20 bits (it’s ironic that Meridian Audio responsible for that research also helped to create the failed Pono Player). For video we should be near the limit as well, so the video bandwidth should stop increasing while communication capabilities keep on improving—thus making the need for better compression less important.

So we got no men, we got no money and we got no reason too to keep developing the new formats (beside niche ones and ego projects like the very fresh Quite Okay Audio). As usual, I’ll be glad to be proven wrong.

This entry was posted on Friday, February 3rd, 2023 at 7:19 am and is filed under Useless Rants, Various Video Codecs. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.