Archive for December, 2022

FFhistory: Fabrice Bellard

Friday, December 30th, 2022

The first time I encountered his work was (then unknowingly to me) when I found out that INSTALL.EXE from Sierra quest was packed with LZEXE. And a couple of years later while looking at all the packages some Mandrake Linux distribution I’d encountered a package called ffmpeg that could transcode various newer formats (it was also the time when I got a first DiXV 3 movie rip and the only players I had could play MPEG-1 at best). And I’d kept encountering his name in random places…
(more…)

FFhistory: prologue

Monday, December 26th, 2022

Before talking about people in history of FFmpeg and libav it’s worth making an outline of the whole history. After all, personal histories are related to the global events in the project.

As you know, FFmpeg was started by Fabrice Bellard under a pseudonym Gerard Lantau exactly twenty two years ago. Why a pseudonym? Probably because back in the day it was too easy to get sued by patent holders but I may be wrong. Initially it was hosted on SourceForge (which had an affectionate nickname SourceForget thanks for the various outages and failures it had) but after several years it moved to a different hosting (somebody said that after the move the traffic to SourceForge dropped by third).

And thus the era of FFmpeg in MPlayer began (it ended for good rather recently when after different perturbations Árpi could not provide hosting and some Bulgarian provider stepped in). And it was not just hosting, there were some MPlayer-specific code hacks inside (starting from fastmemcpy to libpostproc). More importantly, MPlayer has provided a lot of developers that have been playing a very important role since—like Carl Eugen Hoyos or Michael Niedermayer. The latter became the project leader as FFabrice stopped contributing actively in 2005 or so, he had other awesome projects to work at.

I consider the time of about 2003-2009 the golden age of FFmpeg: new developers, new features, fast growth and so on. Some people drifted in because they wanted to fix some encoding or decoding problem Summer of Code program participation started in 2006, so various students came in and usually left (very few stayed).

Then the consequences of fame started to sink in: FFmpeg became a recognized solution used in the enterprise so more developers were coming with patches to support various enterprise stuff (like broadcasting-specific containers) and disagreements with the project management style were rising (like migrating to Git, having regular releases instead of sticking at 0.4.9pre1 and—the unthinkable—throwing out some outdated features). So the split happened and after the period of initial confusion libav was born.

The following period was a war of FFmpeg against libav, no points for guessing which project won it (I think mostly to the efforts of Carl Eugen, hence I call the project of this period CEmpeg). But in the course of it changed a lot in order to win more popularity with the Linux distributions (and to be able to merge changes from libav).

Yet this was the beginning of its end. As I said before, now the development is mostly bugfixes and updates for various wrappers (both for hardware and software codecs) plus filters (do we really need another gstreamer?). A certain Jean-Baptiste from VideoLAN project undertook a valiant effort to reform FFmpeg by introducing a clearer structure for resolving technical arguments and tried to finally get proper funding from those large companies using the software but so far jbmpeg seems like a mostly failed effort to me, both from the financial and the organisational point of view.


Now with all this context it should be easier for me to talk about the roles of individual developers and other people whose work was important for the project (like compn or Piotr Bandurski) as they would fit somewhere on the timeline and not act in vacuum.

End of history

Sunday, December 25th, 2022

So libav.org is no more (and probably nobody has noticed that). From what I heard the server was essentially a dead weight so the provider (who hosted it for free) did not want to keep it running for nothing. And with this the story of Libav ends for good. And while FFmpeg/MNmpeg/CEmpeg/jbmpeg (the naming should become clearer in the course of the following posts) seems to have active development, it’s essentially dead too. Remember, dinosaurs went extinct not because of a single catastrophic even but because they’ve been phased out gradually for their inability to adapt. Originally FFmpeg was one of the tools to support popular formats in opensource (but who remembers libquicktime or avifile nowadays?), slowly becoming the tool and then turning less and less relevant as the multimedia zoo of formats people care about shrunk down to essentially two families (MPEG and Xiph+Baidu), video decoding and encoding being done in hardware and newer codecs supported in jbmpeg not natively but in the wrappers.

This seems to me as a good point to start a series of posts about different people in FFmpeg and Libav history and their impact on it, both good and bad. It may be not complete or unbiased but it should serve at least as a starting point for those curious enough and show it was not just Fabrice Bellard and Michael Niedermayer responsible for the greatness of the project but there were many others as well—and they all had various deficiencies that affected the project in a bad way too. To give an example, how many of you remember the close ties between MPlayer and FFmpeg? How many of you remember such names as Alex Beregszaszi or Justin Ruggles? How many of you know which of three Mikes of the project reverse engineered Windows Media Video 8, which one contributed just small patches and which one is responsible for the test system?

So while I’m still remembering at least some of these details, I should write it down before it’s lost to the winds of time.

What does it matter?

Friday, December 23rd, 2022

This question is a translation of russian “какая разница?” (which literally means “what is the difference?” or “is there a difference at all?”) but it has two appalling meanings in russian.

The legend says that after russian empire occupied Western Ukraine (probably in XVIII century), some Ukrainian representatives came to the official governing the region and asked for giving Ukrainian language a higher status, for example make official announcements in Ukrainian. The governor asked them if they understand russian language and after they said yes he asked what does it matter then which language to speak. Since those times the question is used as a marker of russian chauvinism (and of course a narrative that russians push): why should local languages and culture matter if the colonised nations understand russian?

The other meaning is for the everyday use. In this case it means “this thing is trivial and not worth discussing, so let’s forget about it”. Of course it’s often used to evade the blame or at least an inconvenient question as in “—Who broke the window? —What does it matter, we should think about replacing it.”

So why I’m writing about it? A third meaning to the phrase seems to appear and it does not serve their agenda this time. I follow the news related to the war and a good deal of them are about russian officials making inane claims, saying outright lies and spitting completely unscientific bullshit. And usually their newer statements contradict previous ones so you either need to have the memory of a goldfish or be well-trained in doublethink to believe them. So what does it matter what they say? After all this time the only thing that should be taken into account is their actions and what they said should interest only the judges at the future tribunal.

X is outside politics

Saturday, December 17th, 2022

Yesterday russia had another missile strike on Ukrainian territory, leaving some cities without electricity, heating and water for a day. As expected from a terrorist state. But here I’d like to talk about all those spineless people and organisations that keep supporting russia indirectly by not expelling it because “X is outside politics”.

Nowadays the phrase “X is outside politics” where X is arts, sport, video games or whatever is mainly used as an excuse to do nothing against entities using that thing exactly in political reasons. Kinda like it is made to be outside politics by some natural force and nothing can change it—instead of keeping it outside politics. It is like saying that gas and oil are outside politics and ignoring all the conflicts that have happened in order to control its sources and transport let alone russian blackmail attempts where they drop prices and increase shipped volumes for their allies (like Hungary) and cut it off for the countries not agreeing to their political demands (like Poland or Ukraine).

And yet there are people claiming that other things are outside politics so you should not e.g. boycott russian art just because russians have started genocidal war and in their majority approves it (even if they don’t want to fight in it themselves).

Art

People forget that back in the day creating art was a long and costly process, so works of art were usually commissioned by wealthy people directly (or created to suit their taste in hope the potential patron will reward the creator for dedicating such work in his honour). Of course that meant that such works were often created in a biased way to either flatter the person or to push the favourable narrative. And the creator usually could spin own views into the book (a classic example: La Divina Commedia is famous not only for synthesising various beliefs on afterlife into a single system but also for its author’s views on his political opponents, some being placed in Hell already while they were still alive during the writing of the book).

Now take into account that the state can manipulate the works of art in its own interests—encourage art on a certain theme (like the various scenes from happy lives of Soviet workers and peasants), outlaw and destroy art that does not conform to certain criteria (there are many countries where mentioning LGBT+ themes is outlawed and when such laws are passed a lot of previously “outside politics” works become criminal offences). The work of art may be an abstract thing, but its creator is a human person that lives in a society and has to obey the rules (or suffer the consequences) let alone subtler filter bubbles created by the state (for example, there has nothing happened on 5th of June 1989 at Tiananmen Square, most of Lenin’s comrades were unpersons and Oceania has always been at war with Eastasia). And of course nothing prevents the state from taking those works of art and repurposing them later, giving new context and meaning (as I said previously, almost every russian symbol, starting with the country name itself, is stolen from elsewhere).

Ballet and gymnastics

One would wonder how such rather abstract forms of arts can serve political purposes. Like with sports (which is discussed below), it can be used as propaganda for outside usually pretending on peacefulness (“look, such refined people can’t be bad”) and for inside to unite the people behind something (“look, our performers are recognized in the whole world, we’re the greatest nation”).

Soviet Union used ballet for such purposes to the point that it got a memetic status (“…and as for ballet, we’re better at it than the rest of the planet”). North Korea has a mass gymnastics show called Arirang which is for impressing people how large North Korean collectives act like one person.

Dramatic art and literature

This is probably the most obvious example but Italian and European Commission officials do not understand it.

As mentioned above, plays and books are often written with some explicit political goals (even if they’re a good reading after all those centuries like Jonathan Swift’s), some works simply comprise political views of their authors even if it’s not the main objective (sometimes it’s just a passing remark on a group of people the author dislikes, but it’s still there). And of course the work can be repurposed later.

Soviet Union initially tried to push for the internationalism, the brotherhood of all people and so on, but after the attempts to expand have mostly failed (some countries were conquered but not the whole of Europe or Asia let alone Americas) they switched to the old rhetoric equalling USSR to a new russian empire (i.e. bringing civilisation to the outskirts, being the guide who knows the way to the future and other bullshit) while condemning it (for having a czar and not Polutburo as the ruler). This means that most of pre-1917 russian classics could be used not just to demonstrate how bad was life for a common peasant (alluding it has improved greatly under Soviets) but also the old rhetoric about empire, its strength and defeating the enemies could be used almost verbatim as well. And new works had to conform the so-called party line (the current outlook on the internal and external politics) otherwise it would not be published.

Now let’s look at the other parent of modern russia. Beside creating some of the best propaganda pieces, Nazi Germany has repurposed a lot of past art to its goals. The famous Deutschlandlied (the one that starts with words “Germany, Germany above all, above everything in the world”) was created in times of disunited German kingdoms, duchies and counties and the idea was that they all should unite to one single Germany instead of having petty squabbles between each others—and you know how those words are perceived after 1939. Richard Wagner created his operas because he was interested in the German past and writing epic music with no connection to the modern days (beside maybe some jabs at critics) and that’s why the Bavarian fairytale king supported him—but you know how his music was used in 1930 and what associations it brings now. One of the books by Strugatsky brothers contains the following passage about Nietzsche: “He was a great poet but he was very unlucky with the fans.” And indeed, Friedrich Nietzsche wrote his philosophical works about individual men, how they should cope with the unwelcoming world and strive to become better—so when his idea of superhuman (in moral sense) were adopted by Nazis they still remained outside politics, didn’t they?

Video games

Video games are a work of art too (even if not all) and in many cases they tell a story and have a setting. So some games may push a certain narrative by having a “proper” version of history, “properly” behaving parties and so on. For example, Company of Heroes 2 had hostile reception in russia because you could commit the same atrocities with Soviet soldiers as in real life (like in East Prussia) and in russia they believe that there were no such things (the same way as they don’t want to hear about Bucha or Izyum now).

And considering how russian dictatorship controls all mass media, do you think they won’t try to “convince” game studios (with both money and threats of criminal prosecution) that they should make more “patriotic” games?

Sport

The Olympic games were initially conceived as apolitical competitions (that’s why Greek city-states even stopped fighting during the games). When they were revived in 1896, they tried to preserve the spirit—and then 1936 Summer Olympics happened.

Nazi Germany saw it as an opportunity to demonstrate its power and superiority of its people. In a sense it was the first modern Olympics and not just for the technical details: initially it was a competition between individuals for the sake of promoting sports and chivalrous rivalry. Afterwards it became a canonical show-off for a hosting country and it was state-sponsored teams taking part and not amateurs who trained at their own expense.

Consider the following: in the last decades hosting an Olympics (or often any other large sports event) does not bring profit to the hosting country (especially considering how the objects that had to be built to conform to the newer IOC standards are too large for any domestic use afterwards and have to be either demolished or maintained without a meaningful return). So the hosting country should be a wealthy country willing to throw away money for what? For prestige and show-off. That’s why nowadays large sports events happen in countries with dubious reputation (like Temporary Occupied West Taiwan, Qatar, russia or South Africa).

And you should remember the doping scandal for 2014 Winter Olympics—if the government used its state security in order to organise mass cheating at doping, do you think it hadn’t treated winning at the games as political goal?

There’s another fun fact: for both russia and Temporary Occupied West Taiwan the majority of males in their teams are not merely people training at the state expense, they are also on active military duty. There may be many possible explanations but some state politics is involved there as well.

So you may see all those sports events as apolitical but certain countries do not—and they act accordingly.

Conclusions

I hope it’s clear by now that “X is outside politics” is merely a suggestion and not an inherent property. Something stays outside politics only as long as no party is using it for some political purposes.

In my previous lengthy rant I described how I moved away from russian culture even if I’d been exposed to it long and extensively. For other people, who did not grow up in russian colonies unlike me, it should be even easier.

Similarly, still admitting russians to international competitions of any kind is allowing them to keep spreading their narratives both inside and outside the country (at least russia has thought in advance and bought enough people in various sports committees to keep allowing them even if not under their own flag).

And if you’re still asking why—believing russian narratives is exactly what made this war so horrible, long and devastating (and sadly not for russia). And it enables other parties (like Temporary Occupied West Taiwan) to influence your decisions as well. After all, convincing your enemy that he’s lost already is a victory indeed.

Ignoring feedback as the root of most modern problems

Friday, December 9th, 2022

I might sound as a sidekick to Captain Obvious but some things are needed to be said again and again.

Life in its entirety is built on a feedback loop: if you react on external conditions to maximise positive output you’ll survive; it’s not just evolution, individual organism also would die if they don’t search for food when they’re hungry and don’t flee from danger. But when you have large systems with some resilience, some of the feedback may be ignored for some time. And sometimes all of the feedback gets ignored permanently and that is the start of that system downfall. It has happened before the humanity (dinosaur extinction most likely happened because they were unable to adapt to the changing conditions—because they could ignore the changes until it was too late so they were mostly replaced by more agile animals) but here we’ll review human-made systems (mostly social and political) and failures of those.

Essentially, the cycle works like this: at first a system emerges (or is created) to fulfil some purpose, it grows large, then it starts getting optimised for a target different from the initial one, and when the differences between its initial purpose and the goal it serves grow large enough for that system to face an uneasy choice—get reformed or get eliminated. This happens because the feedback from the wrong decisions at stage three (optimising for a different target) get ignored. So let’s start with reviewing how this happens.

You probably have heard about Goodhart’s law (and if not, it’s a shame) which essentially says that if you optimise something only for a certain metric, that metric will stop mattering. To give a technical example from the area I know: there are various ways to calculate how close lossy-encoded video is to the source and the higher the value of such metric the better video should look to human eye. So far so good, right? But every metric can be manipulated and some researches found that by applying some preprocessing the popular VMAF metric can be improved while the picture quality may feel degraded (it’s not purely theoretical, here in AV1 encoder source there’s some code based on their earlier ideas). Also I heard an anecdote about Korean researchers making an AAC encoder based on neural networks that gave excellent PSNR while sounding like garbage to human ear. Now that you have an idea on how it works, let’s move to less quantifiable things.

I can’t say I understand modern economics well, but essentially in the old days companies were supposed to bring profit by providing a product or service better than the competition (in the best case by being a monopoly). Then the companies became too large to be owned by one guy or family (there are exceptions of course) so the goal became to increase the worth of the stock that shareholders possess—by increasing its market value. Again, initially such stock prices were tied to the easily quantifiable things like assets, income and profit; now it’s more about expectations of growth. Somebody from the past simply would not understand how a company having losses for years still has its stock growing. My cynical view is that it works as long as there are enough people willing to believe that it works (which is also the case with modern economics in general IMO).

Now let’s look at the forms of government. Usually there are no clear goals in either of the systems and people often can’t predict the consequences of their actions (and when they can, they’re not willing to). And there is the usual human tendency to make things better for oneself no matter what consequences it bears for the others. Thus in order to keep system vital you need to get feedback from the people (even if it’s just to know where to send forces to squash rebellion like in today Iran). That’s why China keeps several systems (party members, state security and the state news agency) to report stuff from the regions to the centre. That’s why Winston Churchill said that “democracy is the worst form of government except for all those other forms that have been tried from time to time”. Voting in sane(r) countries is a way to keep the authorities informed about what people want (even if the results of the voting are ignored afterwards). IMO the system could be improved by having a test on the basics (e.g. what parties are out there, what are their governing principles and key points of the programs—just to make sure you make a conscious vote) and account votes of those who passed the test and not separately (in order to do what the concise people voted for and if they’re indecisive—do what the majority wants). But of course such system can be exploited (plus it’s somewhat harder to implement) and we have Gresham’s law in action here (a voter is a voter and any populist would prefer a thousand people voting for him out of tradition to ten people voting because he served their interests but who might not vote for him the next time). The cases of what happens when feedback is ignored will be reviewed a bit later.

Now let’s have a look at social networks. From what I see, originally they were created to attract people by connecting them based on some feature (and make money off them of course). With the time though the goal has mutated to attracting advertisers and selling them audience based on some features. I’m not going to describe what tricks are used to make users behave in the way advertisers like the best but it seems to turn users off those platforms.

As we got to the point where not taking feedback into account has led us to a crisis, there are three possible outcomes: keep ignoring the feedback (aka Soviet Army approach), try to manipulate the metric in hope that it will improve things (shamanism) or do something to the reality to make it conform to the metric (Froogle approach). If you think that taking feedback into account is a possible option too you are an incurable optimist.

Ignoring feedback altogether is a rather simple thing to do. If you don’t like something just call it fake news or enemy propaganda and be done with it. Why I called it Soviet Army approach? Because for me it’s associated with the tradition that was common for the commanders in russian empire, USSR and current russia: keep doing what’s ordered no matter how pointless it is. There are jokes about e.g. painting grass green and asphalt in dark grey to conform the rules, but considering all the variations of it I’ve read in the memoirs and heard from people who served in Soviet Army I doubt it’s merely jokes. Similarly we can see how in this war russians are attacking Bakhmut (or Sieverodonetsk earlier) and how they made Chornobaivka a legend. Denying reality just prolongs the inevitable.

Manipulating metrics is another step, where you hope that by cooking the books and rigging the process you’ll get the desired outcome. As Richard Feynman said, nature cannot be fooled—but what about people? Maybe if we report that the majority supports us the majority will think they’re supporting us because everybody else does? This starts with rigging elections and ends with russia where the only feedback taken into account is what the führer wants to hear, so the election results are always falsified to have the ruling party get the “convincing” majority of votes, opinion polls are treated by population as “the authorities sent me to ask if you’re satisfied with everything, yes or yes?” and it is widely speculated that only good news about the war get to the führer-in-the-bunker (unless they’re too hard to hide, then they’re greatly delayed as it happened with their cruiser, fleeing Kharkiv region or Kherson). In theory this approach may work in some cases where you act in a system powered mostly by human beliefs and you have factual information yourself, otherwise it causes more damage than the previous case.

And finally we have the case where you not merely reject feedback from reality but rather try to force reality to give the feedback you want. I called it Froogle approach after one story (sorry, can’t find the link): when this search service for products on sale was still in development, the team encountered a case where a certain irrelevant product (a garden gnome) was showing up in the results for completely different things. No matter how they tuned their algorithms, it kept showing up—until one day one guy from the team went and bought it thus removing it from the search results. This is also reminding me of the final chapter of The History of One Town by Saltykov-Shchedrin where a satirical depiction of Arakcheyev sees a river on plains that does not fit into his world view and decides to remove it by filling it with rubble from the demolished town (he wanted to build a military-agricultural colony on its place anyway). As you can imagine, it worked for about one day and then the river found its way again. The famous whipping the sea by Xerxes the Great probably belongs here as well. Depending on the resources you have (and against what you fight) it may work out or not—but it’s fun to watch from afar (from close distance though it’s a horror).


We learn from the mistakes (preferably not made by us) and feedback is a very important part of that learning. Pain tells you that what you’re doing is hurting you and you’d better stop doing it, that’s how children learn not to grab hot objects and pointy things. Sadly nowadays in many cases you don’t suffer from the consequences of your mistakes (because the society values individuals a bit too much to allow that) and in some cases it leads to tragic results. Even worse when it happens to the whole countries (because Wesphalian sovereignty is an international counterpart to humanism).

In 2008 russia started a war on Georgia and what was the reaction? The famous EU report that stated essentially that the conflict happened and they don’t know the reasons behind it and can’t verify the claims of participating parties. A year later Obama government started with resetting the relationships. As the result it empowered the aggressor to do more of the same in the future (1938 called and sent its greetings from Munich).

In 2014 russia started essentially the Third World War (I wrote on this opinion of mine) by occupying Crimea and fighting in the East Ukraine using puppet “people republics” (like soviet russia did in 1917-1922). What was the reaction? Essentially it boiled down to “bad russia, don’t do it again—and you, Ukraine, should reconcile” (plus a couple of rather formal sanctions to show some concern). And in the same time russia tried to use its gas as a leverage (not for the first time though and not for the last). Baltic states took the hint and made their systems independent from russia: there were LNG terminals built and new legislation for controlling gas suppliers was passed (like requiring minimum storage fill level). Germany did nothing and as the result it met the second phase of war with near-empty storages (belonging to russia) and no means to transport gas but the pipelines from russia (mostly owned by guess which country)—all against the Directive 2009/73/EC. To me this alone is enough to start investigating Merkel (and preceding) government for high treason, but we all know this is unlikely. The lack of punishment for acting against national interests and in favour of another country will encourage others to do the same (see Scholz and China). And there’s still an issue of Hungary and its well-paid affection to russia…

The situation in the USA is not much better as both parties do not want to recognize that people might have a reason to vote for their opponents and it’s not just russia/China/big tech meddling with the elections. And instead of thinking what can be done to win more voters the parties mostly pass the blame and try to turn their views into a small religion. Also I find the calls to disregard the laws because the other party “cheated” rather disturbing. Thanks to this other countries see America as a laughing stock and, more importantly, withering power and expect to occupy Ukraine, Taiwan or Syria without repercussions.

I want to end this rant with a passage from Going Postal that resonates with me:

You see, I believe in freedom, Mr Lipwig. Not many people do, although they will of course protest otherwise. And no practical definition of freedom would be complete without the freedom to take the consequences. Indeed, it is the freedom upon which all the others are based.