Kostya's Boring Codec World

Time from time I’m asked that question and since people can’t see why I’m not going to design a new format (even though the reasons are obvious) here’s the answer. Format in this context means both codec and container.

There are too many of them already. And they suck in different ways. And I believe it’s impossible to make format that will appeal to everybody so it will suck in some aspect. Either it will lack some features or will be too extensible that it will impose too much complexity on implementation. Lossless codecs are often written in such way that they require a special container because not even Matroska can encapsulate them properly. Lossless video codecs all offer about the same compression level and it’s law of diminishing returns in action (exponentially more time on compression yields only single percent of compression gain at best). Intermediate codecs sacrifice compression gains to speed. Advanced codecs are often some standard ripoffs (e.g. if progression keeps, VP11 will be based on H.266 but with multiple alternative reference frames and their peculiar binary coder). Containers suck either at complexity, compliance or flexibility. And there’s Ogg.

It is hard to write good tools for it. I have written some encoders and what I have:

1. Zip Motion Blocks Video encoder (palettised) — I might be the only user;
2. IMA ADPCM QT encoder — noone cares;
3. M$ Video 1 encoder — got a nice review in 2009 and was merged as is into FFmpeg in 2011 just because. Probably noone cares about it either;
4. AAC bitstream writer — it sucked so much that many talented people who tried to improve it afterwards just gave up and never returned to it again;
5. ProRes encoder — for some reason it become popular and made me realize that noone caring about your encoder is a good thing.

Writing an encoder for a new format requires a lot of testing and tuning (especially for audio) and that requires both hardware and time which I lack. I had enough fun with AAC.

It is very hard to get adoption for the format. See previous two items. You should have good tools to interest users and there are too many different formats already to compete with. These are not the times when people were so desperate that they’d accept anything that was opensource and somewhat fulfilled their wishes (like Vorbis despite it being not hardware decoders-friendly and bundled with Ogg, or Matroska despite it being Matroska).

So, I shan’t develop a new format because it will take a lot of time from me with extremely little chances that results of that work will be ever used. Pity that lossless codecs creators didn’t think about it.

Finally I’ve found some time to play with i.MX6-based Utilite which I intended to use as a home box for various stuff (like running fetchmail, irssi, simple web server etc. — in other words not desktop). So here’s a quick review:

• does not work with my display (1920×1200, DVI input)
• does not allow logging in via SSH (it refuses passwords) and the same problem with sudo later
• does not have IPv6 enabled (not a grave problem but my provider has moved to IPv6 already)
• serial port works as good as telegraph in magnetic storm (honestly, it gives all type of characters on terminal except the ones you can read let alone want, typing one character per minute is somewhat better)

I might be really old but this is not a development board (at least it’s positioned by desktop) so I expect it to work. And unlike previous product by the same company one cannot blame it on hardware — it’s i.MX6, not Tegra2.

So I’ve ordered Cubietruck already (I have Cubieboard2 at work and it has been running fine right from the start).

P.S. Raspberry Pi can go to hell.

This is a translation of this page by unknown author. It’s rather old but recently I remembered it for some reason and decided to share.

Day One

A hacker comes to a diner and sees that salt shaker can be opened by anyone and anything can be put inside. The hacker comes home and writes a letter to the diner manager: “I, meG@Duc, have found a salt shaker vulnerability in your diner. A malicious person can open it and put poison there! Please fix it!”

Day Two

Diner manager gets that mail along with other correspondence and shrugs: “What an imagination”

Day Five

The hacker comes to diner and puts some poison into every salt shaker. Three hundred people are dead, a criminal case against manager is closed after three months because there was no crime from his side. The hacker writes a letter “see now?”.

Day Ninety Six

Manager orders special salt shakers with a combination lock. Diner guests feel that they don’t get something.

Day Ninety Seven

Hacker discovers that holes in salt shaker pass salt in both direction (and other substances too). He writes a letter and pisses into ever salt shaker. Three hundred people don’t come there ever again, thirty people went to hospital with poisoning. Hacker sends manager a SMS “How d’ya like it?”. Manager spends three months being interrogated and a year on probation.

Day 188

Manager vows never to work at any diner ever again and be a lumberjack instead. Engineers are working on one-way salt shaker design. Meanwhile waiters remove all old salt shakers and give salt on demand.

Day 190

The hacker steals a salt shaker and researches it at home. Then he writes a new letter to the manager: “I, meG@Duc, stole a salt shaker and find this outraging! Anyone can steal a salt shaker from your diner!” So far abstinent manager goes home and drinks vodka.

Day 193

The hacker discovers that all salt shakers are chained to the tables. He talks about his achievements at the next hacker conference and receives an award for protecting society and customers’ needs. At least manages doesn’t find this out.

Day 194

All hackers from the conference make a devious plan. They go to the diner and take all salt from shakers. meG@Duc then writes another complaint about low customer service and that anyone can deprive everyone else of salt.

Thus a new salt shaker design is needed. Engineers are working on it while waiters still give salt on demand. Manages goes abroad and uses room service only — no cafes, bars or restaurants.

Day 200

Customers discover that in order to get salt they have to call waiter, show their ID and get special 8-digit one-time code for a salt shaker. Repeat the same for pepper.

It’s pretty obvious but I got requests to write this nevertheless.

All known containers suck, some of them suck gloriously, some of them plainly suck. And there’s Ogg Matroska Ogg.

There are several features that distinguish container usefulness:

• flexibility (supporting various codecs and number of streams);
• easy to parse;
• well-defined specification (there must be a format with such thing);
• metadata support;
• low overhead (bytes needed to define frame size and other properties);
• advanced features for insane people.

Now let’s review containers grouped by design.

Raw or raw with header. Those are the simplest and codec-specific. Besides being designed (usually) for only one stream and one codec, they often decide to save bits on frames and in result you have hard time implementing seeking (say hello to FLAC or Moosepack SV7). Some have seek table at least (old Monkey’s Audio has two — for byte and bit position).

Your favourite FLV belongs to this category — it has one audio and one video stream with no headers (and that’s why it has its own flavour of VP6 with frame dimensions stored at every frame) though one can abuse it to add a data stream. And of course some Chinese used it to store HEVC too in the stupidest way possible (for starters they have introduced half a dozen of different video codec IDs for it).

Chunk-based. The most popular category that refuses to go away. The best representative is RIFF (M$ ripoff of EA IFF format, there are many specific RIFF variants known — AVI, RMF, WAV, WebP) and runner-up is MOV/MP4. AVI is verily the pinnacle — flexible, extensible, every frame is its own chunk. What can go wrong with it? The usual thing: abuse. Too many idiots implemented their own AVI writers with whatever bugs they could introduce and it got even worse when codecs started to employ B-frames. Intel worked around by adding combined I+B-frame and dummy frame afterwards so decoder would handle it internally (you can see it both in Indeo 4 and their I.263). DiVX on the other tentacle… And variable framerate is not for AVI either (unless you simply use zero frames to define skips).

As for MOV/MP4 there seems to be a problem with parsing custom atoms (there are too many atom types around). And of course you have nice abuse like ASF packets stored inside MOV packets if you use Flip4Mac.

And if you replace chunks with an unholy mix of tags and UIDs you get MXF. That format doesn’t have a specification but rather a swarm of them so you don’t know which ones you’ll need to demux some file.

There’s NUT — probably the only format out there with two specifications and three or four implementations, each not agreeing with all other.

MPEG-TS inspired. MPEG-TS is one of overengineered container formats that nothing in this world would be able to demux a TS file with all possible features. And forget about seeking (unless you have an external index or build index yourself).

Of course such design inspired a lot of other formats that have some features of it but often those features are used without understanding why they are there. But result is good for streaming!!!1one

There’s ASF with crazy GUIDs for everything and fixed packet size (which means there’s no direct correspondence between ASF packet and stream packet anymore).

And there’s Ogg. Read this if you still haven’t.

Matroska. That’s a cancer — when you design container that should be able to contain everything and support any feature possible and it gets out of control you get Matroska. It’s based on binary XML, it can have any feature. And it stores every codec in its unique way — see what they call codec specs. So they save bytes here and there and demuxer should put them back, which is not nice, especially if you believe that demuxers and decoders do not need to know about each other.

If you wonder why I haven’t mentioned RealMedia, it’s because this format is an unholy mix of all categories:

• Old RealAudio is rather simple raw + header;
• RealMedia in general is chunk-based format (with a hack for B-frames even).
• Video frame can be split into several packets or several frames can be merged into single packet, a lot like MPEG-TS inspired formats.
• And they had mangled audio streams long before Matroska was here. Actually only some audio codecs data is stored as is, the rest is XORed or has permuted subpackets.

For the last few years I feel some disappointment with my work building up and now I try to explain why.

What kept me working on FFmpeg and later Libav?

Money? Well, I admit that it brought me ~$20000 during all those years of work and it was very helpful in my student years but it’s not that much really and wouldn’t be enough for living even in Ukraine.

Of course the main driving reason was fun of writing code and joy of being useful. I still remember being proud for a week for this commit. I still remember how it was fun (sometimes) to reverse engineer a codec and warm feeling when it’s done. I remember users thanking for the work done and asking for features.

Where did all go?

The project matured and now the situation got different. Previously you mostly had millions of clueless users asking how to transcode something to FLV that were tiring but easy to deal with, now you have more enterprise users that use our code often without acknowledging or contributing back (in the old times Picsearch gave us a database of audio and video files in Internet that used codecs we didn’t support — that’s one of the most valuable contributions ever). But that’s not what kills the fun, “security holes” do.

With an advance of automatic fuzz tools it’s easy to generate millions of damaged files that crash your decoder and yet there are no tools for generating correct patches. Fixing those crashes is tedious, requires a lot of thinking (should I disable it? will it affect decoding correct files? etc.) and in other words not fun at all. You have to balance between having decent code, ability to handle corrupted files and being robust — and in order to account for all possible corner cases in the code from the very beginning you should be more paranoid than FFmpeg leader. And somehow you cannot avoid it, you’re expected to fix it or else. This is like you’re on maintenance contract but without any form of compensation, you just get a mountain of corrupt samples and “have you fixed it yet?” every week. Or you get some “security vulnerability” reports with the same effect. I repeat, this is not fun — so why should I do it for free?

There is only one exception around called VideoLAN. Those guys really show (and not only show) some care and they give back to us. Just in my case I gave them all they wanted and I could give them.

As for the rest, world domination is not my goal, I don’t see fun in maintenance and noone is paying me to do it. Why should I continue?

So I’ll try to finish whatever projects I still have around and end it all. I’ve been around for 9.5 years after all, that’s long enough.

P.S. Maybe I should move to Oljonsbyn.

Sweden is awesome country and its food is plain but decent and heart-warming. And everything gets even better in late December.

There’s no Christmas in Sweden, they have Jul — it’s less Christian in nature but significantly better. As an Ukrainian I totally approve Swedish version because the national Ukrainian animal and products from it are well celebrated.

Yes, that one

Sweden has really good dairy products (I’d especially recommend cheeses and filmjölk), surprisingly good selection of meat (including horse, deer, reindeer and elk), outstanding herring (really, nothing beats fried fresh strömming) and other fish products. And oh so many variations of candies everywhere. And drinks.

Berries

In jul it all gets even better since special food appears — traditional julskinka, sylta (and especially julpressylta), marzipan pigs, special versions of prästost and other cheese… And of course julmust.

One can try traditional game of mine — try different variations of everything. For example, drinks (Trocadero, julmust/påskmust or filmjölk), cheese or even köttbullar (I’ve finally found köttbullar made from deer meat for example). Or at least look at designs of marzipan pigs, every konditori makes their own.

Fun fact — in Gävle they like unofficial symbol of the city so much that they have marzipan goats along with pigs.

And another fun fact — there is a drink from Norrland called Portello and it has unlicensed clones. The real Portello is produced by The Norrland Brewery (aka Vasa Bryggeri) but some other breweries have drinks with similar taste but different name. Guttsta Källa produces Ortello and Mora Bryggeri produces Candelo, though it’s a trollish brewery that makes Rio Cola with familiar Cuba Cola design, too lazy to produce Julmust and Påskmust (see picture) and less popular drinks like Guldus or Haiwa are all renamed too.

Mora Bryggeri — no julmust or påskmust

The Reference Drinks

I’ll continue exploring the wonderful world of Swedish food at every possible occasion.

And some fun things to try:

• julmust with semlor (they are always available in Norrland, it seems);
• fried gravad lax;
• filmjölk with berries (I mix lingon, blueberries and raspberries — they are available frozen all year round);
• julskinka and Wastgöta Kloster cheese on tunnbröd;
• Swedish apples;
• chewing candies;
• and of course Trocadero from any brewery in Norrland!

I still don’t have Internet connection at home (for two months and counting) and not sure if I get any this year.

So here’s the list of rants I’d have written, had I had access:

• SD cards as modern floppies — similarly ubiquitous and slow;
• Swiss cheeses — some are OK, most are slightly sticky, coated in something very smelly and named “chas”;
• WMV9 pre-RTM P-frames decoding: how it should be decoded and why our decoder is still wrong sometimes;
• Bink2 decoder progress report (since I’ve not started working on it yet).

Do not stay tuned.

When I visited VDD13 (they’ve finally made it right — with Trocadero and surströmming, hopefully they’ll keep the level in the future) I could make some additional observations on the rail that finally lead to this post.

First, I like to talk about catenary constructions. They usually come in two variations — a single post with a special support for the wires or two posts with a horizontal construction between them that supports the wires.
It might be hard to believe but they can be æsthetically appealing too.

The top on my list is Sweden (and Netherlands since they seem to employ the same construction). The poles are made from lattice and thus are nice and horizontal supporting constructions are always made as trapezoids.

Runner-up is Switzerland — they also have nice lattice constructions but they seem a bit compressed to me.

Germany has the same poles but for wider ranges they usually have only a wire between poles from which the wire-supporting constructions hang.

Most of the other contries have simple round masts or H-shaped beams that are not interesting, though I must admit French ones have nice wire support constructions reminding of violin bow.

And on the very bottom of the list is Denmark with its large ugly ?-shaped masts in the colour of rust.

And now to the second thing — toilets. I usually try to avoid them but sometimes I feel I have to visit one onboard. So here’s a comparison of that essential thing.

Ukraine — toilets on so-called “InterCity+” should not be that bad, toilets in older carriages are better not be visited at all (and since they dump contents onto tracks directly, toilet rooms are locked long before stations and after them).

Germany — on InterCity trains toilets are decent (or at least tolerable), on ICE they are too small even in the first class. Even on regional trains and trams it seems to be bigger.

France — on TGV first class they are even smaller that on ICE, in the second class even a person smaller than me has problems fitting inside.

Sweden — those people really care. The spaciest toilet rooms on trains I can remember (especially on Reginatåg).

Switzerland — the toilet on Rhätische Bahn regional trains seemed quite good even if those are narrow-gauge trains.

Moral of the story — Swedish trains and railways are the best (and if you have doubts you’re reading the wrong blog).

Of course this interests nobody but I wanted to rant about it for a long time.

General principles:

1. Compact size — I like to be able to fit all of my computers on the desk, any size comparable with power supply unit size would do. Laptops are fine too.
2. Silent — no damned fans.
3. An ability to use normal storage, not 16MB SSB soldered onboard.
4. No x86 CPU.
5. If it’s a laptop it should be able to work for 10 hours with battery.

Display:

1. 4:3 aspect ratio. If displays nowadays are made for movie-watchers then it’s a sad world. Too much of vertical space is eaten by various toolbars, menu bars and such.
2. sane resolution. Again, 1920×1080 may be ideal for movie-watchers but I prefer it to be either VGA-based (i.e. multiple of 640×480 or 800×600) or power of two based. And whoever thought about 1366×768 should burn in hell!

Performance — if Libav compiles in ten minutes on dual core system then it’s fast enough for me.

ARM-based laptops are almost good for that, especially performance wise. There’s just one big “but” — they are almost all are for Android or chromebooks. And Baidu has never intended those systems for any real usage. Playing games — fine, browsing — passable (though Firefox 3 on my old PowerPC MacMini with 512 MB RAM gives much better experience than Chrome on tablet with 1GB RAM), editing texts (code) — absolute fail. I can live without a numpad on keyboard (it’s a legacy for accountants and their calculators after all) but not having even “delete” key (there’s only backspace) is pathetic.

So I live with a faint hope that there will be a computer good enough for me.