Bad audio quality - preparing audio data for transfer over audio track

[yrucrem]

Hello!

I'm having trouble with the audio quality sent over an audio track and was hoping that someone could provide some tips.

The audio data comes as float values in the range [-1, 1] from the source, but to send them over the audio track I need a vector of bytes, and i suspect that the way I am converting the values might be the problem.

This is the setup of the audio track in a nutshell:

	rtc::Description::Audio audio("audio", rtc::Description::Direction::SendRecv);

	auto track_name = "audio-stream";
	auto opusCodec = 96;

	audio.addOpusCodec(opusCodec);
	audio.addSSRC(audio_track_ssrc, track_name);

	std::shared_ptr<rtc::Track> track = peer_connection->addTrack(audio);
	auto rtpConfig = std::make_shared<rtc::RtpPacketizationConfig>(audio_track_ssrc, track_name, opusCodec, 4800);
	auto handler = std::make_shared<rtc::OpusRtpPacketizer>(rtpConfig);

	handler->addToChain(std::make_shared<rtc::OpusRtpDepacketizer>());
	handler->addToChain(std::make_shared<rtc::RtcpReceivingSession>());
	handler->addToChain(std::make_shared<rtc::RtcpSrReporter>(rtpConfig));

	track->setMediaHandler(handler);

Now when sending data I get a vector of float values and create a vector of std::bytes like this:

	int8_t d = static_cast<int8_t>(0x7f * data[i]);
	std::byte b = *(std::byte *)&d; // Reinterpret the bits as an unsigned value.

On the other end of the connection I do the opposite to get float values again. The resulting audio is understandable but very distortet and croaky sounding.

I also tried converting the float values to 16 bit integers and creating a vector of bytes from that, but while it sounded different it does not at all sound better.

Does anyone have some tipps or hints want I might be doing wrong here? Or what is a good way to transform float values in the range [-1, 1] to std::byte values, so that they are suitable for sending via Track::send()?

Thanks in advance for any input and best regards,
Patrick

[paullouisageneau]

If you specify the format as Opus, you have to send Opus-encoded audio data (by the way, the clock rate should be 48000, not 4800).

If you don't want to encode as Opus, you can send the audio as PCMA as you can perform the encoding on your own:

rtc::Description::Audio audio("audio", rtc::Description::Direction::SendRecv);
audio.addPcmaCodec(payloadType);
audio.addSSRC(ssrc, name);

std::shared_ptr<rtc::Track> track = peer_connection->addTrack(audio);
auto rtpConfig = std::make_shared<rtc::RtpPacketizationConfig>(ssrc, name, payloadType, 8000);
auto handler = std::make_shared<rtc::AudioRtpPacketizer>(rtpConfig);
handler->addToChain(std::make_shared<rtc::AudioRtpDepacketizer>());
handler->addToChain(std::make_shared<rtc::RtcpReceivingSession>());
handler->addToChain(std::make_shared<rtc::RtcpSrReporter>(rtpConfig));
track->setMediaHandler(handler);

Samples must then be captured at 8KHz and encoded to G.711 PCMA.

[yrucrem]

Hello!

Thank you for the PCMA/PCMU vs. Opus summary and the tip about the SFU example, that sounds like a very good start, I will dig into that as soon as I can.

Best regards,
Patrick

[yrucrem]

Hello!

First of all thanks again for your support, I have made good progress using the Opus codec. The voice connection works, but I still have two, probably connected, issues:

1. The voice quality is mostly good but too slow and therefore too deep.
2. There is a noticable delay of about 1 second until the other person hears audio.

I have the connection set up as described in the first post, and transfer the packets as soon as they become available from the microphone (after encoding with Opus). I made sure that all involved components work with a rate of 48000. Is there anything on the side of the WebRTC connection that could cause the slowdown and the delay?

As always thanks in advance for any input you can offer!

Best regards,
Patrick

[paullouisageneau]

There is no buffer for media by default, so it can't induce delays. This looks like a timestamp issue, or a stereo/mono confusion (Opus is stereo by default, for mono you need to set it explicitly with addAudioCodec(pt, "opus/48000/1")).

[yrucrem]

Hello!

Thank you for the tipp! I will test this as soon as I can and give an update if it works.

Best regards,
Patrick

[yrucrem]

Hello again!

Unfortunatelly I am still having trouble with the audio quality when mixing multiple audio tracks. I am aware that this is not exactly the purview of libdatachannel I thought I still ask if you maybe have some experience using the Opus codec, because I thinkt that this is the root of my current problem.

What I am doing is collecting the audio packets for each participating peer connection and when I have a packet for each one I do the following:

1. Decode the audio data for each peer using opus_decode_float().
2. For each peer, mix the decoded audio data from all other peers (adding them up and making sure the result is in [-1, 1]).
3. Encoding the mixed audio data for a given peer and sending it.

I don't think that there are any timing issues, because if I don't send the mixed data, but for example the original data of another peer, everything is comprehensible. When I send the mixed data, the audio is very distortet.

Do you maybe have any tipps for working with the Opus codec and WebRTC connections? Any code examples would help, but I was not lucky so far, finding examples close to my setup (C++ using libdatachannel).

As always thanks in advance for any input or advice!

Best regards,
Patrick