Kostya's Boring Codec World

Today I’d like to talk about how opensource projects are supported by “community” on example of FFmpeg/Libav.

Obviously I’ve chosen this example because I know some facts about internal politics and the fact libavcodec is de facto multimedia decoding standard used on every platform and by most multimedia processing tools out there.

What good did it bring to the project(s)? Some fame but that’s probably it—the largest users don’t even bother to acknowledge in public that they use it (cough, BaidUTube, cough). There’s an enormous amount of code (that serves as a good compiler suite too) but it’s maintained mostly by volunteers and people who have to use it at work (as they were hired because they worked on it in the first place). That’s it: the best material gain is an employment because you’ve showed off your skills or you can take occasional consulting work with varying quality of tasks and pay. Some people are paid to improve or write new decoder. Some are hired to work on improving protocol support and hardly paid at all (true story). Some simply sigh at yet another “please implement this for my app” mail. That’s good but where are the money to pay for task the project itself (e.g. refactor old horrible code, add tests, implement new feature or fix some old problem)?

There was an attempt to set up a foundation to gather money and use them for the project but it didn’t work well even before the split and got completely derailed after; and of course it was not a good idea to set it up in the USA since IRS refused to recognize it as non-profit organisation as other open-source projects have experienced (including X.org). The best part is how much money it could raise—I don’t remember the actual sum but it was relatively low like less than $20,000 (please correct me if I’m wrong) and it came mostly from caught (L)GPL violators IIRC.

Let’s take a successful opensource project that uses libavcodec, that would be VLC. For example, last VideoLAN Developer Days definitely costed tens and tens of thousands ${proper_currency} — for accommodating about hundred of people, reimbursing (at least some of) their travel costs, food etc etc; the event was sponsored by the largest Internet advertising company, the largest French advertising company and the largest French VideoLAN advertising company. I remember talks that they could even employ one developer full-time. And would VLC be any useful without libavcodec? And while they probably are the biggest opensource supporter of FFmpeg (they host their Git repository after all) and their developers write some code time from time they hardly do anything else—there is a bounty program but it’s a complete joke since it lists mostly done tasks nobody will claim reward for (and despite me pointing them to that fact nothing has been done). And obviously it doesn’t have tasks that would be beneficial for FFmpeg/Libav but not for VLC directly.

Let’s take a look at some commercial user. All those video hosting sites are available mostly because there’s enough bandwidth to stream video and because there is free code to support most of the formats uploaded by users so they don’t have to bother about details. And that free code is… (no points for guessing). And the biggest video hosting site was mentioned above. They don’t provide any support nor even acknowledge that they use certain opensource projects. Of course one can point to Baidu Summer of Code program but it’s for involving students into opensource (and probably finding fresh meat for themselves) and it doesn’t work that good for projects—you mostly have students willing to get money and/or credit for résumé. They tend to do the task and disappear completely. I’m sure that projects would prefer to have financial freedom instead to sponsor specific tasks that could be undertaken by anyone (not just students) and last whatever time is needed (not just summer). If the project gets some support from large companies it’s usually because a developer from that project working in that company convinced the management to do so.

Even worse is the situation with distributions because they tend to demand free technical support from you: fixing your own bugs, fixing other projects that use your code and such. What do you get in return? Nothing. The recent Debian and Xscreensaver messup is not some special example, it’s too typical.

It’s funny how the best sponsor for Libav might be Lu_minem.it—a small Italian company whose founder is Libav developer and thus knows what the project needs.

How do I fit into all of this? I was FFmpeg and then Libav developer, later a multimedia related work has found me because of my work on decoders (I’d been trying to find a good job myself but failed and had to accept the best offer I got). I was around core developers of the projects and thus could learn the facts written above. What have I got beside developing experience, acquaintances and pessimistic outlook on life? BSoC 2006-2009 participation (where I managed to finish only one project in time really despite the formal passing of it) plus some smaller sum of money for rewriting a component of swscale to make it LGPL. And some free dinners from VideoLAN foundation. So I’m fine but seeing that the project cannot afford paying a developer for some internal project (like writing a new RealMedia demuxer) is still very sad.

Warning: if you do not recognize names mentioned here you might be too young.

I’ve been using computer for about nineteen years and during that time I tried various players for various formats. My curiosity for internal format design made me search for information about compression methods, source code for decoders and such. So it led me to the current state (doing nothing). Yet for the many multimedia players I know there are some naming issues and that’s what I want to talk about.

My first versatile player was PLAYSND.COM by Yuri Tumarin. This 13kB DOS program could play a lot of various sound formats like WAV, VOC, MIDI variants and Adlib tracker music (RAD, HSC and lots of other variants). The best part is that it played some compressed WAV files too (various ADPCM variants and more). Excellent tool but the name is too bland and hard to search for.

Speaking of Adlib tracker formats, there’s an opensource player with Adlib emulator supporting lots and lots of them. The problem with it (beside being outdated now)? It’s called adplug. A good name to be blocked by a generic rule!

Let’s move to video players.

My first Linux player for VideoCDs was MpegTV. It was a commercial program but again, it was a country where nobody bothered about piracy and it was the only player on Linux I knew that could decode VideoCDs without stuttering. The player was doing its task fine but its name is rather cringeworthy.

Then I found out about DVD-oriented players like Ogle and Xine. Good names. And I still use Xine sometime when I need to play DVDs.

And the golden standard of multimedia on Unix systems—XAnim. The only bad thing is that the last time I checked it didn’t work correctly in 32-bit X11 mode. But it did its job well and I’m still grateful it exists (and also its binary plugins were good binary specification for missing codecs).

And there’s MPlayer. It was fast, it had many useful features and codecs supported (I still use it as a testbed for running some 32-bit VfW/DMO codecs when I cannot write a decoder without debug) but its codebase was horrible (in some cases legendarily horrible) and the name is both bland and reminds of mplayer32.exe (which crashed and hanged a lot too).

And one of its forks is named after one of the horrible chunks in libavcodec and its author operates under pseudonym. So was it really worth it to name the player MPV?

And I conclude this review with a well-known multimedia player that I won’t use. When I think about VLC the first meaning coming to my mind is variable length codes. And when the only good thing about your player name is the number of puns you can make I’d use something with more decent name thank you very much.

So it has come to this, some events and discussions made me write this post.

How I imagine the perfect process for new codecs? It’s rather simple model: you have some places where ideas and enthusiasts swarm and from their work and selecting best ideas new candidate codec is born.

There are such places for all codec types: audio enthusiasts can find testers at Hydrogenaudio, video enthusiasts can talk at Doom9, general and image compression people seem to be present at encode.ru. In first approximation it works as expected—people propose ideas, test new compression programs and report benchmarks, suggest improvements. What can be wrong there? Just one thing: people making software incompatible with anything else (custom containers/archive formats) and trying to push it on everybody. After you invent some format make sure it works in some standard environment (for compressors it’s usually single file compression mode, .tar.xz seems to be more popular than .7z even if they use the same LZMA algorithm; for codecs it should be the standard container—even Matroska would do). And document the format too—properly instead of usual “bug off” level.

There are standardised codecs that undergo similar process: various companies or researchers submit their work, a base for a new standard is chosen, new proposals try to improve it. And then companies start to push their patented shit there and that’s where the system goes wrong (QMF in MPEG Audio Layer III anyone?). It’s not better when some company tries to push its product as a standard without any evaluation (and thus we get wonderful line of SMPTE VC-x codecs for instance).

And there’s OggXiph. This is again a community that designs codecs mostly because they can and pushes them mostly because they’re Free™ and OpenSource™ and they mostly suck otherwise: Ogg format is for streaming not good for anything, most people still don’t know that it’s Ogg/FLAC because it was developed outside (and has horrible raw stream format), Speex has no readable specification and easier understood with disassembling the library rather than reading source code, Theora is an outdated enterprise grade code, Opus has its issues (but it’s rather good, one cannot deny that), Daala will probably never happen.

And what do I see in recent news? Alliance for Open Media plans to release first draft of their codec soon and it is:

• hosted on baidusource.com;
• for now just libvpx with some names changes;
• everything else about it screams Baidu too.

It if looks like Duck, produces codecs like Duck and has the same source code as Duck, then it probably is DuckOn2Baidu.

At least in the old times there was some competition of ideas in codecs so one could choose between different codecs giving good results—and in some cases they were available for various ecosystems too (e.g. Indeo was present in AVI and MOV, ClearVideo managed to get into AVI, MOV and RM). Now it’s just foam of lossless codecs that even their authors forget about next year and one or two companies pushing their stuff on everybody. And that makes me sad.

It’s time to say myself: you are no longer relevant. All remotely useful codecs have been reverse engineered already and most of them are obsolete. Everybody uses either H.26[45]+AAC or VP{3,8,9}+Opus and nothing else is required (even by VLC). And I grew tired too if it wasn’t obvious from my previous posts (I don’t even blame lu_zero anymore).

And thus my plans are to document Duck TrueMotion 2X and VP4, ClearVideo and hopefully VX.

In NihAV (yeah, like it’ll ever happen) I decided to implement just fringe codecs like those and no new modern codecs maybe except Bink2 and WMV3 (I know what should be done to support beta P-frames there but the libavcodec decoder is so unwieldy that my brain switches off trying to analyse how to do that change there; before you say anything it’s a part of the code written before I took over it and failed to make it into full-featured decoder during GSoC 2006).

So, this FOSDEM certain Vittorio gave a talk about reverse engineering codecs, all materials are here. Here are my thoughts about it.

First of all, I gave a talk about similar topic once at VDD 12, it was my first and last public talk. Of course it was a fail and there was no single question asked (and mind you, VDD attendants are usually know multimedia much deeper than ordinary FOSDEM visitor). So I think it takes a lot of courage to give such talk so Vittorio did a good job here.

Now to the remarks about the presentation itself.

He calls himself a pupil of Kostya (slide 2) — the name is not rare so I don’t know which Kostya he meant. Definitely not me as he’s yet to show any signs he has learned something from me.

Slide 6 mentions examples of rip-off codecs and gets it wrong too. Real had some licensed codecs, RealVideo 2 is a licensed Intel rip-off of H.263 (and surprisingly Real had even one original codec, lossless audio one). Oh, and the VP family starts with VP3, before that it was Duck TrueMotion 1/2.

Categories mentioned on the next slide are rather random subsets of all video codecs out there. I’ve REd several codecs that do not belong to either category (one of them was used in Hedgewars clones called Worms BTW).

TDSC part of the talk is remarkable for two things: 5-line tool (slide 16) is something I write from heart when I’m too lazy to find the previous version of it and it actually took less time to RE it than to talk about how it was done (the main issue there was to call JPEG decoder inside TDSC decoder).

As for Canopus HQX — I’ve written about it years ago. Beside the profile-specific tables there’s actual decoding to be done (you know, VLCs, DCT, that kind of stuff). But large tables in binary specification fend off reverse engineers quite often.

So, it was a good introductory talk but I haven’t missed anything by not attending it.

Everybody knowing even a bit about me knows that I live in Germany and I’d rather live in Sweden. One of the reasons is that Sweden feels like future (i.e. improved) version of Germany, and here I finally explain why.

Railways. While railway system in Germany is mostly fine it still can have some improvements: train carriages should have an electric outlet per seat. Now it’s fulfilled only in ICE 1st class seats without neighbouring seats. Also hassle-free (or simply free) WiFi onboard. Also it would be nice to have tilting trains like X2000 because it’s all fine and good when you have dedicated high-speed line but quite often you don’t and it would benefit more to have higher speed there (quick example: route Mannheim–Saarbrücken takes 1:32 by Regional Express with its 6 stops or 1:19 by ICE/TGV with only one intermediate stop). And Sweden is more liberal in a sense that now you have line Stockholm–Göteborg serviced by both SJ and MTR expresses (compared to that France where they mark every railway station as belonging to SNCF feels like Ukraine).

Service. Sweden has supermarkets open every day (shorter opening hours on holidays of course) and they also contain local post office too (here post offices are usually selling stationery and some goods from local supermarket instead and share the space with Postbank instead). In Germany almost everything is closed on Sundays. Also it’s funny how in Sweden Lidl can be considered low-tier supermarket (Hemköp/Willy:s, ICA and Coop are much better), in Germany Lidl is considered middle-tier supermarket and in Czech Republic it’s considered one of the best. And there’s much better recycling: all plastic bottles are accepted by the same machine (here you bottles that should be recycled elsewhere and bottles that are not recycled at all), there’s printing on packaging where it should sort to (i.e. hard/soft plastic, paper, metal etc) and water quality is way higher too.

Religion. Sweden definitely doesn’t bother with Christianity that much (and that’s why I enjoy visiting Swedish churches). And as a nice touch even in the centre of Stockholm you have squares named after Odin and Thor and a street called “way to Valhalla”. In Germany they still cannot admit that Wednesday is Wotan’s (or Odin’s) day.

Government offices. Sweden has got it right—census matters are handled by the tax office. Germany is yet to realize that idea. One can point out that in the USA citizenship and taxpaying are the same thing (since you have to file a declaration in any case and pay taxes even if you live and work abroad) but the execution in this case is botched.

P.S. I blame lu_zero for not giving reasons to blame him in this post in the usual way.

So it has come to this. Looks like I should not eat the best Sweden can offer—namely, meat (especially game) and seafood (especially herring). Well, what can I eat in Sweden then?

Not surprisingly there are still many good and tasty things for me. Let’s see…

Chicken. At least in Stockholm any decent supermarket offers warm grilled chicken—full, half or just legs. And each supermarket offers its own brand of chicken sausages to grill.

Dairy products. Beside my favourite cheese and filmjölk varieties there are still many nice and crazy things, like youghurt-quark or drinkable quark (or ordinary cottage cheese that goes well with blueberry-raspberry jam).

Bread and pastry. Sweden has much better bread variety than Germany—that means more different kinds of bread, especially hard bread (if you’re German feel free to stay offended anyway or go discuss beer with Czechs). I also like västerbottenostpaj—a pie with Västerbotten cheese. And semlor. And various Swedish cakes. I should note that Swedes like to use cinnamon and whipped cream in various dishes and rare Swedish cake has no whipped cream. When I was staying at an hotel in Örnsköldsvik (or simply Ö-vik for locals) it has a large bowl of whipped cream among other things for breakfast. And I guess if I simply sprinkle some cinnamon onto a lump of whipped cream it will make Swedish Minimal Cake (I’d like it for sure). And of course marzipan pigs in season.

Crisps(chips). You have it under both names and they are extremely tasty. And when Swedes get bored with potatoes they can make any root into chips, including the ones I’ve never eaten in other form. Here in Germany I bought a packet of chips only once and it was Svenska LantChips of course.

Snacks in general. They usually occupy half of my bag when I return from Sweden (the rest is mostly drinks and other stuff). Wonderful nut mixes, dried fruit and berries, various confectionery things. And of course all supermarkets (beside lidl-iest of them) have naturgodis section where one can mix various sorts of these (think of loose candy stand but with dried strawberries, roasted almonds, peanuts in chocolate and such). And loose candy stands with at least dozen varieties are really everywhere.

Fruits and berries. Paradoxically I can buy vendetta oranges in Sweden but I’ve not seen them here in Germany—Sweden must be closer to Italy then (BTW I still blame lu_zero for not recognizing short name for Sicilian blood oranges despite pretending to know a lot about citruses). There’s wider selection of fruit than in Germany. And there’s much much better selection of berries. You have traditional raspberries, blueberries, black and red currants, RIMs, strawberries and also Nordic berries—Swedish blueberries (they are smaller and have more intense taste), lingon, cranberries and cloudberries. In Norrland in season you can get even more. On my arrival in December I bought some cherries in supermarket—they were available both fresh and frozen (and it’s hard to buy even frozen cherries in Germany outside season). And I like cherries, especially sour ones (BTW I blame lu_zero for being a competition).

Drinks. Sweden has the best water in the world and one can enjoy drinking tap water there (unlike tolerating tap water in Germany and doing it at your own risk in Ukraine). And that’s why a lot of drinks are sold as concentrates that you have to dilute yourself in proportion 1:3 or 1:6. And because it’s Sweden you have good selection of drinks based on berry juice. I especially like blueberry-cranberry and blueberry-raspberry drinks and of course lingon. Carbonated soft drinks are the best in the world too (and if word Trocadero doesn’t tell you anything you’re reading a wrong blog) but I wrote about it many times.

And of course julskinka. I’m Ukrainian after all.

Last year I’ve said some harsh words about some Danish påskmust. Just in case I was wrong I decided to recheck it and bought another bottle. Here are the pictures:

In case you don’t see the list of ingredients it contains:

• carbonated water;
• elderberry juice;
• blackcurrant juice;
• sugar;
• grape juice;
• malt extract;
• spices;
• caramel colour;
• hops;
• citric acid.

And it tastes like POSIX julmust from AIX brewery would taste—it has right ingredients but also lots of things normally not found in julmust and the taste is weird in result. Only people with perverted taste like Danes (remember, one of them invented C++ and is not ashamed of it) or lu_zero(whom I blame for his tastes quite often) would like it.

Last week I’ve been attending a fifth Libav coding sprint in Pelh?imov. Here’s a report from the host of the current sprint (she was amazing, many thanks). It was fun indeed (some fun provided by Lufthansa canceling my flight because of strike).

It’s the third sprint I attended and there’s a pattern in them. Even if they are called coding sprints there’s not so much coding going there, it’s more discussing various stuff and food than actual coding (though Anton spends a lot of time fixing some decoder usually). As I’m no longer a Libav developer, I simply hang around and try to provide enough trolling and proper drinks, sometimes even sharing a bit of knowledge if anybody wants to listen.

Another recurring theme is AVScale, a saner replacement for libswscale. It gets discussed during the sprints (since Summer 2014) but nothing substantial gets done. The only things we got so far are my proof of concept implementation (I’ll present it later in my post about NAScale, it’s the same thing) and something hacked by 1-2 Italians under influence of alcohol in a couple of hours (with great comments like //luzero doesn't remember this) that has Libav integration but no functionality (my code is complete opposite of that — standalone and doing some useful functions). Well, just wait for new posts about this, they’ll appear eventually.

In general I attend sprints just to see people and new places and have some fun. Among the places where sprints took place — Pelh?imov, Stockholm and Turin — I’d say Stockholm was the best so far as it is the easiest to reach and has the best food to my taste (plus thanks to SouthPole AB hosting more people than at any other sprint). I should attend a sprint again some time.

P.S. I still blame lu_zero for not writing anything about it yet.

I like a good read with analysis of random stuff. The problem is that the planned series of posts on parallels between social organisations and software development are not written yet.

The first post would be dedicated to the idea supplanted by bureaucracy. This happens when somebody has a good idea like “let’s create X foundation to promote good thing Y” and with time it turns into monopoly that dictates the rules to everybody else in that area. A special attention would be paid to the ways such organisations maintain “democratic” façade while making all decisions in private — like with introducing many almost useless contributors into voters and convincing them to vote in the way organisers want. An addendum about how such non-profit organisations get their incredible amount of money would be good too. And if you’re still in need of an example — think about any sports organisation like IOC or FIFA.

The second post would be about cultists — not people adhering to some religion but rather people blindly following something and not even thinking that other people might have other views and needs. For example, iUsers or those who program in Oberon dialects (that seems to include Go despite it being not a direct descendant). The main problem with them is that those cultists usually force their fetish as the only proper solution and the main answer to “but I want to do that” is “it’s not needed and I’ve never did it myself”.

The third post would be about a common tactic switching from peculiar details when advertising stuff to blanket statements when defending from criticism (or vice versa). Among other things it’s very common for Oberon and systemd advocates in a fashion like “Feature X? We have it right in Y. Y sucks? Well, it’s just a single dialect/module, the whole system is wonderful, stop attacking it.” It would be painful but giving parallels between this and terrorists/peaceful Islam would be proper too.

And of course I blame lu_zero for not writing this.