Archive for May, 2025

Bloat is a choice (but not necessarily yours)

Friday, May 23rd, 2025

First of all, I’d like to remind that the term optimisation comes from mathematics and means selection one of the available options in a certain set to maximise certain function. Of course the term was misappropriated by computer “scientists” in the same way as physicists did with biological term plasma and the first association people have with that word is “something that modern games lack”.

And the crucial part in the original definition is that there’s a metric present by which the choice is judged. So even software may be optimal, it’s just the goals set during its creation are different from what you expect (and most likely set by the marketing team).

Hence you have the first kind of bloat: creeping featuritis caused by the desire to earn more money for the product by making it more appealing to the wider masses. Often it does not matter if most of those features are unpolished or even non-existent, as long as the product is bought you can keep lying that they’ll be fixed/improved with the next update or in an upgraded version of the product. I’m still old enough to remember the times when this was called Korean cell-phone syndrome (after electronics manufacturers from a certain country notorious for such tactic) instead of being normal product cycle.

Other kinds of bloat come from the trade-offs that everybody has to make. For instance, there’s sediment accumulating from the previous features that were hard to remove or even to refactor. So it’s there and may break things if you try to remove it (as other components rely on it in unpredictable ways). The most obvious example of that is human genome, what humans create on macroscopic level is usually much much more effective (there’s no need to list exceptions).

And finally, it’s rather conscious sacrifice of efficiency for some other goals. For example, car engines can be more effective in, say, kilometres per litre of fuel consumed but they won’t allow you to accelerate or won’t pass ecological standards (which is a problem with diesel engines, even Volkswagen had to admit it eventually).

The same applies to developing programs: you may write a program without superfluous features, it will require minimal storage and RAM, and it will run extremely fast—but you’ll end up writing it somewhere next century. And it’ll be easier to write it anew than make a port for another system. Also let’s not even think about debugging.

That’s the story with the low- to high-level programming languages. They offer more and more composability and convenience for being slower. You can write program in machine codes but you have to calculate all jump and call offsets by yourself. That’s why assembler was created—and also introduced macros which allowed to not repeat the common sequences by sacrificing a bit of efficiency (since sometimes you could’ve omitted or improved an instruction or two that were not needed in that particular place). But it’s not good enough when you have to port your program to a different OS or even a different CPU, thus C was invented, earning a nickname “portable assembly” (which is no longer true thanks to the standardisation committee but that’s another story). With C you can write a program not only that you often may compile and run on another system, but also modular enough that you can have a comprehension of what it does without going insane after looking at one huge unstructured file. Of course this also has its cost, both in compilation time and efficiency losses that even optimisation passes can’t compensate.

In the same time there were high-level programming languages, and they had a radically different approach: “we don’t care about the details, we want this calculation to run on the machine and produce results”. The main advantage of C++ over C is that it allows better encapsulation of different objects and types (so you can e.g. define your own matrix class and operations over it writing more natural mat_a + mat_b instead of mat_c = matrix_add(mat_a, mat_b) and worrying if you passed the right types). Of course this does not come for free either (virtual tables for class methods take space, you know) but it’s often a small price to pay (especially if you’re using a language that has actually learned something from C++ mistakes).

The same applies to the reusable components aka libraries. You save time on writing code (and sometimes on debugging it as well) while sacrificing speed (since those components are not tailored to your specific use case) and space (because they often bring in dependencies that you don’t really need; web development is full of egregious examples of this).

At least here it is your choice and you decide how you’re going to build your project what you can sacrifice for what goals. With other apps forced onto you it’s not the case.


In conclusion I want to give an example from my field. My hobby multimedia project written by me from scratch does not strive to be the fastest (except occasionally when it gets the fastest open-source encoder or decoder—but that’s for the lack of competition) and I work on it time from time mostly to make sure I understood some concept right; when I try to make something faster it’s usually to make it fast enough for my practical needs. And there’s Paul M. whose project I haven’t advertised enough (here’s the link just in case)—he took the existing codebase and added or improved some bits, striving to be better than the competition (and often successfully). You can give arguments why either approach resulted in bloated software, but I’d say that as long as we’re happy with the results it’s not bloated in either case. It’s optimal in the original meaning of that term, you just haven’t considered what metric was used for picking that solution.

na_game_tool: yet another bunch of formats

Sunday, May 11th, 2025

So, what’s been added since previous report?

  • Road Avenger BIN—as mentioned in an update to the previous post, done;
  • Reaper FMV (from Rollcage game)—decoding works fine but the reconstruction does not because of imperfect IDCT implementation on my side (and the fact they don’t use any kind of clipping to make sure the result will stay in range all while employing different tricks like converting double to int by adding a large constant, storing the result as double and read the second half of it as an integer). Overall, I decided not to count it and threw in a couple of easily REd formats described in the next entry;
  • a pair of other BIN formats from Data East—they’re the same as Road Avenger BIN but not even compressed;
  • Ecco the Dolphin LMS—another raw-tile Sega CD console format except that tile data is arranged more curiously there (128×128 frame is split into 128×32 strips consisting of 8×8 tiles that are arranged in columns). Also it seems to use implicit grey-scale palette (unless there are some external palettes hidden elsewhere), at least watching those documentaries in this way is fine;
  • Interplay M95—it’s been described on The Wiki since 2006 (not by me of course) but nobody bothered to implement a decoder for it until now;
  • Psygnosis SMV (from WipEout game)—I looked at it back in the day but of course it’s been on The Wiki since 2012. Codec 2 was not described completely right there (partition codebooks are stored in the end, after block indices and not before; and it was not immediately clear that block palettes are stored in global frame order and not in partition coding order) but nothing that a bit of debugging can’t fix.

In either case I’ve filled self-imposed quota of a dozen of originally REd formats and started to pick other formats to support (three done, about nine to go). I have noted about three formats worth supporting and if I can’t find more good candidates—well, maybe there are some more low-hanging Sega CD formats…

And as usual, for more supported console formats, both simpler and much more complex, there’s always librempeg (and that’s not talking about other things it offers like the countless audio and video filters).

P.S. Since I don’t want to do a separate rant, I’ll mention some tangentially related things here.

First, a search for more SMV samples on discmaster returned no unknown real SMV samples but some PMV samples with .smv extension including one that could not be decoded (because of zero root chunk length—I’ve fixed that) and a couple that had nothing wrong with them but were not detected as Zork PMV at all for some reason.

Second, I actually used discmaster to check if there are some interesting ARMovie files. Sadly it still lack an option for omitting supported files so I searched for acornReplayAudio files and filtered out the sources containing actual ARMovie files without video track (somehow I don’t want to check all of ~6000 files so if one or two files coming from same CD are audio-only I excluded it from the further search). After all this winnowing I’ve found two samples with Eidos Escape 102 codec. It’s simple and I’ve REd it some time ago so finally I’ve managed to write a decoder for it. At least I don’t have to search for the samples by the magic string like I did back in the day… Sadly there’s no good way to detect old TCA format so I’m not even going to attempt looking for it.

Democracy and open source

Wednesday, May 7th, 2025

Since I have nothing better to do, I decided to finally write a rant about between democracy and open source movement.

I believe democracy is a flawed form of government—because people are flawed. There are two main problems here: first, the fact that people are mostly ignorant and don’t think about the decisions implemented by their representatives; second, people don’t often vote how they want. Yes, there’s a difference here—in the former case people don’t have an idea what their representatives do (nor care about the fine details of the legislation passed) and vote for somebody for such profound reasons as “I voted this way last time” (or even “we’ve always been voting for this party”), “at least it’s not X”, “just for fun”; in the latter case people may know what they want from the candidate and yet keep voting for the wrong person consciously. The reasons in that case may be even sadder: “I want to vote for a winner” (i.e. voting for the candidate who’s most likely to win because that candidate is most likely to win), “that guy controls the main business in our town so we have to vote for him”, “that guy gives out freebies” or “all media claim that’s the best possible candidate ever”. Side note: I’ve heard all the excuses listed here and not just in the recent news about USian elections. Essentially it boils down to two things: people either lacking control (over the information, or even over their own income; the mostly forgotten distributism movement had a lot to say about it) and people not caring about it at all (but still voting for some reason). And it seems to me that open-source movement is a lot like this.

Initially Free Software (free not as in “free beer” but rather like in Stallman’s speeches) was targeting the audience that understood it, namely the programmers and hackers that values the software freedoms and could exercise all of them (yes, including modifying source code and compiling the library/program). Nowadays though there are many users that actually do not care about the software they use as long as it solves their needs (and if not, they’ll either look for an alternative or start pestering the developer to fix it for them, instead of doing it themselves). Like with democracy, people are so used to its presence that they not realise why it matters and don’t care if something happens to it.

The second aspect is the lack of control. I develop software in the old way: I make it useful for me and provide the sources for the curious zero people who may do with it whatever the AGPL license permits. But most developers have to play by the rules of infrastructure providers in order to get their software noticed. In an exaggerated form, if your project is not on GitHub and does not have at least a thousand stars then it does not exist.

And if you got your software popular and included into some kind of distribution or package repository, that means kowtowing to the distribution (or package repository) maintainers. As one of the core libav developers I could observe the interactions between that project and certain Linux distributions. All I can say is that with the recent USian Securing Open Source Software Act and European Cyber Resilience Act they essentially get the same treatment as the developers got from them (but at least it’s more formal).

If you wonder why either is important in the first place, the answer is freedom. Without democracy you have no way to affect what’s forced onto you, without open source you have no chance of getting software for your needs instead of the vendor’s needs (which are usually diametrically opposite to your needs—like having full control over your hardware, wallet and all personal information they can siphon out of you and sell to the highest bidder).

If you wonder what can be done about it, there are two obvious solutions. People may actually start to think about what they’re doing (or choosing) and collect all available information beforehand. And not such utterly improbable solution involves global (thermo)nuclear war—no people mean no problems (or at least survivors will be more occupied with surviving instead of competing who has the latest iPhone model). The chances of the latter are rather good, let’s do nothing and wait.