Kostya's Boring Codec World

Apparently there’s some work been done recently on the ProRes encoders in jbmpeg with the intent not merely to fix the bugs but also to leave just one. So why not talk about the format support, it’s about as entertaining as the recent story in the project demonstrating once again that most of the problems in open-source development can’t be solved by throwing donations, even moderately large ones.

So, ProRes, the format with its history being a rather good demonstration about the project issues (CEmpeg and libav back then, and FFmpeg throughout its history in general). This tale is full of whimsy (depending on your sense of humour of course) and contains moral as well.

Let’s start with the basic facts: ProRes is an intermediate codec from A**ple (since the people tried to avoid attention from that company it’s better to follow the suit and not mention it directly) i.e. an intra-frame codec with moderate compression intended for saving space for raw footage used in editing videos (and some cameras may encode their input into ProRes directly) but not for the end-user consumption. It was introduced in 2007 and its support remained an often requested feature for FFmpeg until the opensource decoder finally appeared in 2011, and opensource encoders followed shortly after. Apparently A**le also released ProRes Raw in 2018 which is a different format but which seems to re-use the same coding principles as the original ProRes. The chances of open-source support for it are very low though.

The initial stage (from the introduction of the format and until its opensource support) took four years. And judging from the copyright dates, it took at least a year to reverse engineer it (probably from PowerPC disassembly). Moral of this part of the story: there are not very many people doing reverse engineering and the work takes time and so no matter how often you request it to get supported (and even offer donations) it may not get done.

Then there was a time known as The Split when FFmpeg was split into libav and what I call CEmpeg (and there was a special fork called FFmbc existing since at least 2009, it will also play a role in this story).

The opensource ProRes decoder was a co-operation between Maxim Poliakovski, a talented reverse engineer known for his work on Indeo and ATRAC codecs among other things, and Baptiste Coudurier, an expert in various broadcasting technologies and the sole developer behind FFmbc fork. The details have never made it to the public but apparently Baptiste decided to rush things and make his version of their joint work available in CEmpeg under GPL and using Elvis Presley pseudonym. I suspect that GPL was chosen as a move to make the interested parties pay to get it re-licensed under more permissive LGPL. Also since it was the first commit under an obvious pseudonym since a long time it created some fun of its own (more about it later).

And a bit later Maxim published his own version of that code under LGPL which was a bit slower IIRC but it was LGPLed and eventually the GPLed version was dropped. If you’ve ever wondered why the ProRes decoder in jbmpeg is located in libavcodec/proresdec2.c now you know.

Moral of this part of the story: even if you fail to make money on it at least it may be a good laugh for somebody else.

Now with the open-source decoding present, it was time for the encoder. And it did not take that long for one to appear. The one submitted to CEmpeg was more or less straightforward encoder based on the existing decoder source as the reference. During FOSDEM 2012 we (libav developers) discussed picking it as well and I pointed out that it was far from being good and the author name sounded like yet another pseudonym (it is like somebody would submit a work signed by a common first name Chuck and rather ordinary last name Norris) and, rather foolishly, proposed to write a better one for a crate of Trocadero. And on the way home from FOSDEM I started developing it using an ARM-based ultrabook (Efika MX Smartbook); a bit later the promised crate of Trocadero in bottles of various size came from Benjamin Larsson.

Considering that “Anatoliy” was not active before and has disappeared shortly after his work was accepted, the theory about it being not his true name still stands. I actually wrote a mail to him, asking in private if this is a pseudonym but got no reply.

Speaking of pseudonyms, considering how the initial decoder commit in CEmpeg was pushed without announcement and under pseudonym and the encoder was submitted under what looked as a pseudonym as well (and ProRes encoder in FFmbc is written by Michael Jackson), I decided to join the fun and submitted A**le Intermediate Codec decoder initially under the pseudonym of Evert Taube (and some of his fictional characters, just like in his songs).

And then the real fun began. Because CEmpeg merged everything from libav while complaining about it, they merged my encoder as well and Michael for some reason decided to add Anatoliy’s copyright to my work. His excuse was that since it has the same bugs it must be based on it. I suspect he was talking about the bug in variable-code writing—the part reconstructed from the decoder and where it was easy to make the same off-by-one mistake—but nobody has attempted to reach me to clarify the situation. I believe that was done just as yet another of his excuses to not acknowledge libav in the source code along with “the work on this code started before libav fork existed so it can’t belong to it”. Mind you, I was not against relabelling the files but I did not understand the reason for doing that. But considering the rather recent mail where he lists “people who have subtantial contributions over the lifetime of the project” and omits Luca Barbato (who has more commits and probably more copyrights on files than I bothered to put) I’m inclined to believe it was mere pettiness.

But let’s talk about more technical stuff. As I mentioned above, the “Anatoliy” guy made a drive-by commit (IIRC he also submitted a small fix for something else while his ProRes encoder was reviewed but that’s all) and disappeared so it was up to CEmpeg maintainers to do work on it (bugfixing, updating interfaces and introducing new features). And for the latter part they mostly did nothing until in 2018 somebody bothered to bring it up to date with the features my encoder had by 2014 (different profiles support, interlaced encoding and alpha plane encoding). At least keeping feature-wise on the decoder was easy for them.

And speaking about the features, alpha support in ProRes encoder helped to uncover another issue (beside that I botched boundaries checks there): that alpha conversion in libswscale was far from perfect either (see ticket #8509) and people used to complain about it. There are other fun things like converting from RGB555 to RGB565 causing a complain that full chroma interpolation is not implemented for that destination format (and why would you convert to YUV in the first place?) but it’s not related to this story.

Judging from the recent Clément’s work, I got some other things in the encoder wrong. To my defence, most of the nuances about bitstream flags and feature combinations are mentioned in the specification released in 2015 and by that time I left libav development entirely and had no reason to care about ProRes (let alone wasting money on the document). And speaking about official documents, A**le still discourages users from using third-party decoders and encoders as they may damage your equipment, naming FFmpeg explicitly (considering their own less than stellar history with software vulnerabilities…)

There’s only one bit left to be mentioned, ProRes Raw. From what I know, Paul B. Mahol worked on reverse engineering the format and figured out most of it but considering that he got fed up with the current jbmpeg situation I doubt we’ll ever see it finished (or that somebody else will complete that work).

And finally moral of the story: ProRes as a format is nothing remarkable but the things related to the open-source support of that format show a lot of characteristic details about the project(s). There’s a saying attributed to Bismarck that those who love laws and sausages should not see how those are made; I hope this story demonstrated that this saying is applicable to open-source multimedia as well.

This entry was posted on Monday, February 5th, 2024 at 3:47 pm and is filed under FFhistory, Useless Rants. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.