It sounds like you’re asking for uncompressed audio? That meets all of your listed requirements. 48kHz * 16bit, single channel = 768kbit/s

We support that already, yup. But it never hurts to see if there's something better than that out there.

You can bootleg your own fast lossless codec by doing delta-encoding on the raw PCM to get a lot of zeros and then feed it through an off-the-shelf fast compressor like snappy/lz4/zstandard/etc. It won't get remotely close to the dedicated audio algorithms, but I wouldn't be surprised if you cut your data size by a factor 2-4 and essentially no CPU cost compared to raw uncompressed audio.

You’ve not done this before have you ?

I haven't, but now I have. I took https://opus-codec.org/static/examples/samples/music_orig.wa... from https://opus-codec.org/examples/. Then I wrote the following snippet of Python code:

    from scipy.io import wavfile
    import numpy as np
    import zstd

    sampling_rate, samples = wavfile.read(r'data/bootleg-compress/music_orig.wav')
    orig = samples.tobytes()

    naive_compressed = zstd.ZSTD_compress(orig)
    deltas = np.diff(samples, prepend=samples.dtype.type(0), axis=0) # Per-channel deltas.
    compressed_deltas = zstd.ZSTD_compress(deltas.ravel()) # Interleave channels and compress.

    decompressed_deltas = np.frombuffer(zstd.ZSTD_uncompress(compressed_deltas), dtype=samples.dtype)
    decompressed = np.cumsum(decompressed_deltas.reshape(deltas.shape), axis=0, dtype=samples.dtype)
    assert np.array_equal(samples, decompressed)

    print(len(orig))
    print(len(naive_compressed))
    print(len(compressed_deltas))

giving:

    17432876
    15518973
    12817602

Looks like my initial estimation of 2-4 was way off (when FLAC achieves ~2 this should've been a red flag), but you do get a ~1.36x reduction in space at basically memory read speed.

Using an encoding for second order differences with storing -127 <= d <= 127 using 1 byte and the others 2 bytes (for an input of 16-bit audio) I got a ratio of ~1.50 for something that can still operate entirely at RAM speed:

    orig = samples.tobytes()
    deltas = np.diff(samples, prepend=samples.dtype.type(0), axis=0)      # Per-channel deltas.
    delta_deltas = np.diff(deltas, prepend=samples.dtype.type(0), axis=0) # Per-channel second-order differences.

    # Many small differences, encode almost all 1-byte differences using 1 byte,
    # using 3 bytes for larger differences. Interleave channels and encode.
    small = np.sum(np.abs(delta_deltas.ravel()) <= 127)
    bootleg = np.zeros(small + (len(delta_deltas.ravel()) - small) * 3, dtype=np.uint8)
    i = 0
    for dda in delta_deltas.flatten():
        if -127 <= dda <= 127:
            bootleg[i] = dda + 127
            i += 1
        else:
            bootleg[i] = 255
            bootleg[i + 1] = (dda + 2**15) % 256
            bootleg[i + 2] = (dda + 2**15) // 256
            i += 3

    compressed_bootleg = zstd.ZSTD_compress(bootleg)
    print(len(compressed_bootleg))

    decompressed_bootleg = zstd.ZSTD_uncompress(compressed_bootleg)
    result = []

    i = 0
    while i < len(bootleg):
        if bootleg[i] < 255:
            result.append(decompressed_bootleg[i] - 127)
            i += 1
        else:
            lo = decompressed_bootleg[i + 1]
            hi = decompressed_bootleg[i + 2]
            result.append(256*hi + lo - 2**15)
            i += 3

    decompressed_delta_deltas = np.array(result, dtype=samples.dtype).reshape(delta_deltas.shape)
    decompressed_deltas = np.cumsum(decompressed_delta_deltas, axis=0, dtype=samples.dtype)
    decompressed = np.cumsum(decompressed_deltas, axis=0, dtype=samples.dtype)
    assert np.array_equal(samples, decompressed)

Prints 11593846.

While I also want a low-computation codec that can save space, the historical use cases unfortunately assumes a lot more CPU power to be compensated for a lot less bandwidth, so there's little research in this area, and there's no real incentive to make something like ProRes and DNxHD as if you are editing audio the SSD speeds has been so fast that you'll run into CPU problems first.

Either that or G.711.

G711 is neither high bitrate nor usable for music.

Then use G.722, it works fine for music.

No, g722 is still a wideband speech codec. Its available frequency goes up to 7 kHz. The uncompressed audio this thread began with goes up to 22 kHz. With g722 you're losing most overtones, or even all overtones from the top of a piano. Please don't use g722 for music apart from on-hold muzak.

How is Audio encoding the most limiting factor in a VR project? :o Afaik Opus encoder eats something like 30-50MHz of one cpu core.

It sounds plausible that it's the most expensive thing on the server side, if you have cheap simulation/behaviour and many concurrent users.

But unless it's a non commercial project, the cost shouldn't be a big deal, so it's still a bit strange.

We work on a community-led fork of the dead commercial High Fidelity project. The server requirements are indeed very light except for audio.

Physics are actually farmed out to the clients themselves, it's a bit of a quirky idea, but it actually works if one isn't concerned with accuracy.

I did a prototype of a 3D low-latency server side mixing system, based on a hypothetical 4k clients, @48k each being mixed with the 64 loudest clients.. Using Opus, forced to Celt mode only and running 256 stereo sample frames at 128kbps.. Worked well, using only 6 cores for that workload.. The mixing was trivial, but the decode and encode of 4k streams was entirely doable.. This issue at that rate was 1.5M network packets a second.. If I was to revisit it, I’d look at using a simple MDCT based codec, with a simple Psychacoustic model based on MPC (minus CVD) and modified for shorter frames + Mdct behaviour versus PQMF behaviour, without any Huffman coding or entropy coding.. And put that codec on the GPU.. Small tests I did using a 1080ti indicated ~1M clients could be decoded, mixed and encoded (same specs as above) problem is then how to handle ~370M network packets a second :)

Edit: Had high hopes for High Fidelity, and came very close to asking for a job there ;) Shame it’s kaput, didn’t know that :(

Those are interesting ideas, thanks! I'll have to try and play with that.

High Fidelity the company is still around, but they pivoted multiple times radically. Initially their plan was social VR of sorts. Then they tried to make a corporate product for meetings and such, and gave up on that right before COVID19 hit!

And after that they ripped out all the 3D and VR and scaled down to a 2D, overhead spatial audio web thing. Think something like Zoom, only you have an icon that you can move around to get closer or further to other people.

The original code still lives on, we picked it up and are working on improvements. Feel free to visit out Discord (see my profile).

Apparently RP1 team handle bigger crowd loads through muxing on the server but not sure exactly how that works out for spatial audio there is a Kent Bye Voices of VR podcast discussing how they got 4k users in the same shard.

Why is your VPS server encoding rather than clients? Are you combining talkers together into one source for doing crowds and avoiding N^2 or something and need to reencode after combining?

Correct, server does spatial audio.

It's a community-led continuation of High Fidelity, a dead commercial project. They made their own proprietary codec with excellent performance we can't use and managed to have a couple thousand people in the same server.

I would like to know the answer to this question from dale_glass too.

Replied to the parent

The Bluetooth codecs are all designed to be very cheap on CPU and low latency - e.g. LC3, AptX or SBC.

I'm skeptical, those are almost always going to be implemented in hardware, so the complexity of a software encoder isn't a design concern.

There is some correlation between the cost of a hardware implementation and complexity of a software implementation. SBC is a very simple codec, but AptX and LC3 might not be much better than Opus.

I couldn't find data on CPU requirements for encode/decode versus, say, Opus, but Apple uses AAC-LD for similar scenarios.

If you are OK with moderately high bitrates, you might prefer something simpler like an ADPCM scheme. It's pretty damn easy to implement ADPCM, certainly a lot less math heavy than MDCT-based schemes, and they achieve good quality at a somewhat higher bitrate (I have no data, but I'd guess 200-250%~ish.)

I believe codec2 is pretty easy computationally. The M17 project uses it IIRC, and implements it on an STM32.

That'd be a good choice except for the requirement to support non-speech audio.

LC3Plus or AAC-LD. Although they likely don’t fit the definition of Open Codec.

Vorbis might be a good choice there