FFhistory: ProRes

Apparently there’s some work been done recently on the ProRes encoders in jbmpeg with the intent not merely to fix the bugs but also to leave just one. So why not talk about the format support, it’s about as entertaining as the recent story in the project demonstrating once again that most of the problems in open-source development can’t be solved by throwing donations, even moderately large ones.

So, ProRes, the format with its history being a rather good demonstration about the project issues (CEmpeg and libav back then, and FFmpeg throughout its history in general). This tale is full of whimsy (depending on your sense of humour of course) and contains moral as well.

Let’s start with the basic facts: ProRes is an intermediate codec from A**ple (since the people tried to avoid attention from that company it’s better to follow the suit and not mention it directly) i.e. an intra-frame codec with moderate compression intended for saving space for raw footage used in editing videos (and some cameras may encode their input into ProRes directly) but not for the end-user consumption. It was introduced in 2007 and its support remained an often requested feature for FFmpeg until the opensource decoder finally appeared in 2011, and opensource encoders followed shortly after. Apparently A**le also released ProRes Raw in 2018 which is a different format but which seems to re-use the same coding principles as the original ProRes. The chances of open-source support for it are very low though.

The initial stage (from the introduction of the format and until its opensource support) took four years. And judging from the copyright dates, it took at least a year to reverse engineer it (probably from PowerPC disassembly). Moral of this part of the story: there are not very many people doing reverse engineering and the work takes time and so no matter how often you request it to get supported (and even offer donations) it may not get done.

Then there was a time known as The Split when FFmpeg was split into libav and what I call CEmpeg (and there was a special fork called FFmbc existing since at least 2009, it will also play a role in this story).

The opensource ProRes decoder was a co-operation between Maxim Poliakovski, a talented reverse engineer known for his work on Indeo and ATRAC codecs among other things, and Baptiste Coudurier, an expert in various broadcasting technologies and the sole developer behind FFmbc fork. The details have never made it to the public but apparently Baptiste decided to rush things and make his version of their joint work available in CEmpeg under GPL and using Elvis Presley pseudonym. I suspect that GPL was chosen as a move to make the interested parties pay to get it re-licensed under more permissive LGPL. Also since it was the first commit under an obvious pseudonym since a long time it created some fun of its own (more about it later).

And a bit later Maxim published his own version of that code under LGPL which was a bit slower IIRC but it was LGPLed and eventually the GPLed version was dropped. If you’ve ever wondered why the ProRes decoder in jbmpeg is located in libavcodec/proresdec2.c now you know.

Moral of this part of the story: even if you fail to make money on it at least it may be a good laugh for somebody else.

Now with the open-source decoding present, it was time for the encoder. And it did not take that long for one to appear. The one submitted to CEmpeg was more or less straightforward encoder based on the existing decoder source as the reference. During FOSDEM 2012 we (libav developers) discussed picking it as well and I pointed out that it was far from being good and the author name sounded like yet another pseudonym (it is like somebody would submit a work signed by a common first name Chuck and rather ordinary last name Norris) and, rather foolishly, proposed to write a better one for a crate of Trocadero. And on the way home from FOSDEM I started developing it using an ARM-based ultrabook (Efika MX Smartbook); a bit later the promised crate of Trocadero in bottles of various size came from Benjamin Larsson.

Considering that “Anatoliy” was not active before and has disappeared shortly after his work was accepted, the theory about it being not his true name still stands. I actually wrote a mail to him, asking in private if this is a pseudonym but got no reply.

Speaking of pseudonyms, considering how the initial decoder commit in CEmpeg was pushed without announcement and under pseudonym and the encoder was submitted under what looked as a pseudonym as well (and ProRes encoder in FFmbc is written by Michael Jackson), I decided to join the fun and submitted A**le Intermediate Codec decoder initially under the pseudonym of Evert Taube (and some of his fictional characters, just like in his songs).

And then the real fun began. Because CEmpeg merged everything from libav while complaining about it, they merged my encoder as well and Michael for some reason decided to add Anatoliy’s copyright to my work. His excuse was that since it has the same bugs it must be based on it. I suspect he was talking about the bug in variable-code writing—the part reconstructed from the decoder and where it was easy to make the same off-by-one mistake—but nobody has attempted to reach me to clarify the situation. I believe that was done just as yet another of his excuses to not acknowledge libav in the source code along with “the work on this code started before libav fork existed so it can’t belong to it”. Mind you, I was not against relabelling the files but I did not understand the reason for doing that. But considering the rather recent mail where he lists “people who have subtantial contributions over the lifetime of the project” and omits Luca Barbato (who has more commits and probably more copyrights on files than I bothered to put) I’m inclined to believe it was mere pettiness.

But let’s talk about more technical stuff. As I mentioned above, the “Anatoliy” guy made a drive-by commit (IIRC he also submitted a small fix for something else while his ProRes encoder was reviewed but that’s all) and disappeared so it was up to CEmpeg maintainers to do work on it (bugfixing, updating interfaces and introducing new features). And for the latter part they mostly did nothing until in 2018 somebody bothered to bring it up to date with the features my encoder had by 2014 (different profiles support, interlaced encoding and alpha plane encoding). At least keeping feature-wise on the decoder was easy for them.

And speaking about the features, alpha support in ProRes encoder helped to uncover another issue (beside that I botched boundaries checks there): that alpha conversion in libswscale was far from perfect either (see ticket #8509) and people used to complain about it. There are other fun things like converting from RGB555 to RGB565 causing a complain that full chroma interpolation is not implemented for that destination format (and why would you convert to YUV in the first place?) but it’s not related to this story.

Judging from the recent Clément’s work, I got some other things in the encoder wrong. To my defence, most of the nuances about bitstream flags and feature combinations are mentioned in the specification released in 2015 and by that time I left libav development entirely and had no reason to care about ProRes (let alone wasting money on the document). And speaking about official documents, A**le still discourages users from using third-party decoders and encoders as they may damage your equipment, naming FFmpeg explicitly (considering their own less than stellar history with software vulnerabilities…)

There’s only one bit left to be mentioned, ProRes Raw. From what I know, Paul B. Mahol worked on reverse engineering the format and figured out most of it but considering that he got fed up with the current jbmpeg situation I doubt we’ll ever see it finished (or that somebody else will complete that work).

And finally moral of the story: ProRes as a format is nothing remarkable but the things related to the open-source support of that format show a lot of characteristic details about the project(s). There’s a saying attributed to Bismarck that those who love laws and sausages should not see how those are made; I hope this story demonstrated that this saying is applicable to open-source multimedia as well.

12 Responses to “FFhistory: ProRes”

  1. Paul says:

    Very interesting that nobody is interested in ProresRAW in FOSS. Maybe it is good sign or more probably very worrisome sign.

  2. Kostya says:

    Sounds exactly like “half glass full/empty” situation. Either you cheer up at nobody at FOSS wanting to use proprietary formats or be saddened by users being locked into fully proprietary software workflows and enjoying it.

    I may somewhat wonder what people use for an intermediate format nowadays but so far it turned out to be DCT or wavelets with simple coefficient coding (because speed there matters more than compression ratio).

  3. Paul says:

    I got banned on devel mailing list by one week, you can now make new rant blog entry: FFhistory: Final Death

  4. Peter says:

    Fully Funded MPEG

  5. Kostya says:

    That’s what Stefano said.

  6. Kostya says:

    I still believe that the project is undead so as long as there are enough misguided individuals and companies that use it and try contributing, its un-life will continue. De facto monopoly even in FOSS is bad.

    Also considering how you got a week suspension instead of three days (like a certain other individual) means they value you more (or your words were closer to truth than his).

  7. blue_misfit says:

    What about 12 bit encoding? None of the open source encoders do this AFAIK and it’s very desirable for HDR content using ProRes 4444 / 4444 XQ

  8. Kostya says:

    I don’t know who is going to implement that (definitely not me though). But I think the situation is perfect – the lack of entities willing to sponsor such work is balanced by the lack of developers capable/willing to perform it.

  9. Dennis says:

    Do you have any plans for any GPU or assembler optimizations for ProRes encoder maybe? Its a bit slow even on good AMD machines with plenty of cores. 12 bit would also be nice ))

    Your blog and encoder are gold Kostya. Spasibo!

    Given nature of opensource evolution its miraculous that FFmpeg has no “RedHat Inc” or “Mozilla Corp” on the side. Makes it much less organized and messier, but truer to the open source core maybe??

  10. Kostya says:

    I do not intend to touch ProRes ever again (and my last real GPU was ATI Mach64 so I’m not likely to ever get into GPU-related programming either). And as you could infer from my post, I don’t know if there’s anybody who would work on the features you want.

    As for the project itself, I suppose the lack of “support” from large companies comes from the patent minefield associated with it. So if some large company would try to keep a fork of it, it would probably immediately get sued by D*lby, DT$, MPEG-LA and such. Even funnier, Baidu uses “we don’t support you so you won’t get sued by large companies” as an excuse (makes one wonder with how many conditions their “support” comes).

  11. compn says:

    never liked prores anyhow. if people wanted prores raw they’d cough up a few thousand coins for it. no bounty no development.

    kostya when are you going to work on ffmbc fork ? obviously its the superior fork.

  12. Kostya says:

    I’d rather not touch any of them – it may be a superior fork but still a fork of you know what code with the same underlying problems.