Archive for the ‘AAC’ Category

Spectral Band Replication in various audio codecs

Thursday, July 7th, 2022

While the war continues I try to distract myself with something less unpleasant, recently it was implementing SBR support for AAC decoder in NihAV. And since I’ve done that why not talk about this technology and its varieties?

The idea of saving bits in lossy codec by coding less important high frequencies in an approximate way is not very new, the first case that comes to mind was aacPlus codec from Coding Technologies that served as a base for AAC SBR (and MP3Pro if you remember such thing). It allows to code the upper part of the spectrum very efficiently compared to the normal AAC codec (I’d say it takes 4-12kbps for that compared to 50-60kbps used for the lower frequencies) which allowed to cut audio bitrate significantly comparing to AAC-LC of similar quality. And since it was a good idea, other codecs used it as well but in different format (because patent infringements are fun when both parties employ hordes of lawyers).

So, let’s look at how it’s done in the codecs I can remember (if you know more feel free to comment):

  • AAC SBR (also mp3PRO but who cares)—the original that inspired other implementations. It works by splitting frame into series of 64-band slots (using complex numbers unless it’s a coarse low-power SBR that uses only real numbers), copying lower frequencies into high ones using a certain shape and adding scaled noise or tones (those two are mutually exclusive). For transmission efficiency lots of those parameters are derived from the configuration (that is transmitted once for couple frames) and essentially only envelopes used to shape coefficients and noise plus some flags are coded. You have to generate a lot of tables (like how QMF bands are grouped for four modes of operation, what gains to use on coefficients/noise/tones for each QMF band in each slot and so on). Eventually there were other variants developed (because there are other AAC codecs that could use it) but the approach remains the same;
  • E-AC-3—this codec has SPectral eXtension which divides frame into fixed sub-bands and copies data from lowed sub-bands, applies a specific scale to that data and blends it with noise scaled with another scale;
  • AC-4—this one has A-SPX that looks a lot like the original SBR (and considering that D*lby got the team behind it it’s not that surprising). I can’t be bothered to look at the finer details but from a quick glance it looks very similar (starting with all those FixVar and VarFix envelopes). If you want to know more about the implementation just ask Paul B. Mahol, it should be more fun than the usual questions about AC-4 he gets;
  • ATRAC9 (but not earlier)—this codec seems to split spectrum into four parts, fills them either with mirrored coefficients from below or with noise and applies coarser or finer scaling to those bands;
  • WMA9 (or was it WMAPro or WMA3?)—as usual it’s “we should overengineer AAC” approach. There’s not much known about how it really functions but it seems to split higher frequencies into variable-length bands and code motion vectors for each band telling from which position to copy (and since audio frames in MDCT-based codec are essentially P-frames, this is too close to being a video codec for my taste). There are three modes of operation for the bands too: copy data, fill with noise, or copy only the large coefficients and fill the rest with noise. I have an impression they tried to make it less computation-heavy than AAC SBR while having the similar or larger amount of features.

I guess you can see how these approaches are different yet alike at the same time and why it was not much fun to implement it. Yet I still don’t consider this time wasted as I gained more understanding on how it works (and why I didn’t want to touch it before). Now maybe it’s time to finally play with inline assembly in Rust.

Notes on AAC quantisation

Thursday, March 19th, 2009

I should have written this earlier if not for non-FFmpeg work I have to do here. BTW, are some linguists around there that can explain a relation between bureaucratic and textile (“bureaucracy” comes from a sort of cloth used to cover tables, “red tape” is rather obvious, Russian “????????”, “????????” and “??????????” are also related to a process of obtaining thin threads). Ahem.

AAC coding has two computationally costly operations — MDCT and coefficient quantisation. While the former takes more cycles per one call, the latter is usually called several times for each frame, so those times tend to sum up and outweigh MDCT in bad encoders (like mine). From rate distortion theory we know how to determine proper quantizers for AAC – distortion caused by that quantisation multipled by lambda plus number of bits needed to code that band with this quantiser should be minimal for given value of lambda.

How could we achieve this? Well, use one of three approaches:

  1. Assign some fixed quantizers
  2. Use some ad hoc rule to determine quantiser and then refine its value a bit (aka heuristic, since it gives good speed, it is widely used)
  3. Try all possible quantisers by brute force or Viterbi method (optimal but very slow)

With heuristic you have one catch: if your primary guess on quantiser is not good then refining either takes a lot of time or gives you far from optimal result. Trellis-based search is implemented in my decoder and results in around 20x slower than realtime encoding speed (i.e. encoding one second of audio takes 20 seconds of CPU time) on modern CPUs. I’m playing with something heuristical and fast.

Now to quantising itself.

Each coefficient is quantised as out = (int)pow(in / quantiser, 0.75);. Division of floating-point numbers is slow, taking power of a number is even slower. You can convert MDCT coefficients to the power of three fourths (and quantisers are also converted in precomputed table), thus getting rid of power. FAAC also multiplies coefficients so they are always quantised except for taking integer part. My decoder just multiplies possible codebook vectors by that quantiser and compares it with input coefficients leaving them intact. I also had an idea to present MDCT coefficients in base pow(2, 0.25) making it easy to manipulate but someone still has to test it where base conversions won’t eat all of the gain. I have also tried several optimisations like not trying to match coefficients against all codebook vectors using only close enough vectors. More approaches to try.

(I hope these notes will form “How I Wrote the Best Opensource AAC Encoder Around (to Accompany x264)” memoirs :-S )

AAC encoder and psy model

Thursday, March 5th, 2009

As you may know, I am working (mostly NOT working though :(, but still remember about it) on AAC encoder. This morning I’ve made simpler psychoacoustic model inspired by FAAC (yes, Dark Shikari, FAAC has some sort of hardcoded psy model) work with my encoder.

I’ll try to use this blog with its original purpose — to formalize my thoughts on subject at hand. I thinks many posts on different aspects of psychoacoustics will follow before more or less suitable encoder will appear. “More or less suitable” means it should be at least a good audio encoding counterpart for x264 (while “fully suitable” means total world domination).

Too bad there’s not enough time (always).

AAC: Life goes on

Monday, August 25th, 2008

Well, Summer of Code 2008 goes to end. I think I’ve completed my task – creating simple AAC encoder with psychoacoustics
(here are my application link and a rough task description). I’ve written AAC encoder with psychoacoustic model made after 3GPP TS26.403 and it seems to work fine (on multichannel sources too).

Now there’s a small catch – to be included into FFmpeg it should be optimal in the terms of coding. That means minding rate distortion, choosing optimal quantizers and codebooks, etc. Now it’s a bit harder task, so I think final encoder will be in SVN this autumn or later. Any pointers on information how to optimally encode some part of stream are welcome (for miscellaneous stuff like pulses, temporary noise shaping, etc.).

Stay tuned (and use Flac instead 😉 ).

A bit more

Friday, August 8th, 2008

With low-pass filter my AAC encoder is more or less feature-complete. Of course there’s still more room for improvements but it’s pretty fine now. I’d like to submit it for review but it depends on some parts of AAC decoder and it’s still under review :-(. So I don’t have much to do until then.

So I switched to last GSoC task and hacked again at RV40 loop filter. Well, filter invoking pattern is almost there and I’ve fixed several bugs in actual filtering code. Bit it’s not there yet. Maybe in a month it will be so if AAC encoder won’t take all my time again.

News + Extra

Sunday, August 3rd, 2008

AAC front: to compete with other encoders I have to implement low-pass filter. Benjamin suggested Butterworth filter, so I will try it next week. Hopefully that will be the last big feature to do.

RV front: looks like deblocking pattern is generated from comparing motion vectors, if the difference for subblocks is greater than 3, then edge between them is scheduled for loop filtering. Don’t expect working loop filter implementation too soon though, I still have to deal with AAC encoder and it’s more important.

Extra: I’ve finally decided to buy ASUS Eee, it was easy thing to do – there’s only one model (Eee 701 4G with Win XP installed) for about the same price of four hundred bucks (maybe $450 in greedy shops). So the first thing I did with it was installing Linux and tearing down that stupid “Designed for Windows” label (which was surprisingly easy thing to do and left no marks on laptop surface).

Now here are complains about Ubuntu Eee (I don’t have USB DVD drive and Xandros hasn’t worked from USB flash drive for me): it requires some hacking of system configuration to make it work (like shutdown properly) but that I can live with, but the braindead thing is that gcc is installed (why?) without any development header or library, so you can’t compile even “Hello, world!” program. Both of those issues are resolved, so I just need to make this toy more useful to me 🙂

AAC: encoder progress

Friday, August 1st, 2008

If there are still people interested in my work on AAC encoder (I doubt though), here is a status report for you.

Tasks completed:

  • Make encoder provide lookahead samples
  • Make psy model use these samples to produce block switching decision
  • M/S case handling (at least I hope so)
  • Multichannel coding (from mono to 5.1, seems to work fine)

So there are only two bits left: tune model to produce better sound and commit that all to FFmpeg SVN.

The problem with current implementation is that it follows given bitrate too closely, so some frames may be coded too badly resulting in audible artifacts. I will add ABR mode and quality-based mode to make it better. Another issue is how to use codec options to turn them on, so Robert Swain can add profiles for them ;).

In related news: AAC decoder is slowly migrating to FFmpeg SVN, so I should be able to submit my encoder for review and inclusion as well.

A bit of AAC news

Sunday, July 20th, 2008

I’ve not abandoned work on AAC encoder completely.
For example, I think now my psy model calculates and handles bitrate better. Now the only goals I set to myself are:

  • Make encoder provide psy model lookahead samples for block switching
  • Finetune psy model, or more specifically:
    • block switching decision
    • finish M/S case handling (for now it does not update all psy model information and result in artifacts on several test samples)
    • adjust scalefactors to reduce quantisation distortion
    • some other tricks from 3GPP TS26.403?

After that there are several ways to go: improve quality until it beats other encoders (at least libfaac), implement SBR/PS encoding (the latter is easier to do), introduce multichannel coding.

I suspect that at least one person will suggest to do it all.

Troubles with Psy Model

Monday, July 14th, 2008

I’m battling with psychoacoustic model. 3GPP TS26.403 seemed clear at the start but then problems have begun.

The main problem is perceptual entropy estimation. Since MDCT coefficients differ by magnitude in FFmpeg and 3GPP, perceptual entropy estimated by formulae from 3GPP differs much from actual number of bits to code. Also 3GPP encoder always makes scalefactor lie in range 104…164, beats me why (and its center does not correspond to scale = 1.0 either).

I have to investigate further before continuing work on psy model. I also hope to see AAC decoder in FFmpeg SVN soon and push my work on encoder there as well.

AAC: Nachrichten pro Woche

Saturday, July 5th, 2008

Here is this week portion of AAC-related news:

  • I was working on psychoacoustic model and fixes for it. Now encoder should always produce correct files (i.e. decodable without bitstream errors). Sound quality may be low though.
  • There was a bug in MDCT calculation which resulted in wrong spectrum.
  • My test device for AAC has broken 🙁 Where I can find a decent pair of headphones that won’t break that easily? Especially in this country.

And just in case my mentor’s reading this, here are my plans:

  • Improve and finish 3GPP TS26.403-based psychoacoustic model.
  • Implement block switching.
  • Add sine windows.
  • Sync my encoder with current AAC decoder code (maybe it will be committed by then?)