I’m trying to stream audio bytes to a discord bot, but instead of music there’s a lot of noise. Moreover, the bot connects successfully, the codec works. The first thing that comes to mind is that I either take the data in the wrong place, or I transfer it to the bot incorrectly. Moreover, the bot itself works correctly, it connects to the voice channel correctly, and even recognizes sounds, but not the correct ones. I use Exoplayer to play a local mp3 file on the device itself and in Discord. It plays on the phone itself without problems

Discord Bot requrements

To work with discord bots I use Kord. Its requirements for transmitted data are described in the documentation:

A frame of 20ms Opus-encoded 48k stereo audio data.

Codec settings

Based on this, I set the following codec settings, as per the requirement:

@OptIn(UnstableApi::class)
class BotAudioProcessor(
    private val audioDataListener: AudioDataListener
): BaseAudioProcessor(){
    val codec = Opus()
    val SAMPLE_RATE = Constants.SampleRate._48000()
    val CHANNELS = Constants.Channels.stereo()
    val APPLICATION = Constants.Application.audio()
    val FRAME_SIZE = Constants.FrameSize._960()
    val COMPLEXITY = Constants.Complexity.instance(10)
    val BITRATE = Constants.Bitrate.max()

    init {
        codec.encoderInit(SAMPLE_RATE, CHANNELS, APPLICATION)
        codec.encoderSetComplexity(COMPLEXITY)
        codec.encoderSetBitrate(BITRATE)
    }

    override fun onConfigure(inputFormat: AudioProcessor.AudioFormat): AudioProcessor.AudioFormat {
        val outputFormat = AudioProcessor.AudioFormat(
            SAMPLE_RATE.v,
            CHANNELS.v,
            C.ENCODING_PCM_16BIT
        )
        return outputFormat
    }
    //... queueInput()
}

Here I set all the parameters as requested in the documentation. All these values ​​were set via the codec, the only thing is that I spent an hour calculating the frame size
frame size = SAMPLE_RATE * 0.02 (20ms) = 960

Getting audio streaming

If I understand correctly, uncompressed PCM streams can be obtained using AudioProcessor, for this, I inherit from BaseAudioProcessor and override the queueInput method, in which I encode the buffer in Opus format, and then pass it to the listener, which will send the stream to the discord bot.

override fun queueInput(inputBuffer: ByteBuffer) {
    if (inputBuffer.remaining() < FRAME_SIZE.v / 2) {
        Log.e("BotAudioProcessor", "Input buffer too small")
        return
    }
    val frame = ShortArray( FRAME_SIZE.v / 2)
    inputBuffer.asShortBuffer().get(frame)
    val encoded = codec.encode(frame, FRAME_SIZE)

    audioDataListener.onAudioData(encoded!!.toByteArray())
}

Stream data to the bot

Next, the received audio stream is saved in a variable, which is then broadcast by the bot.

@UnstableApi
class MusicBot(...) : AudioDataListener {
    private var audioData: ByteArray? = null
    //...

    @OptIn(KordVoice::class)
    suspend fun start() {
        kord!!.on<ReadyEvent> {
            //...
            voiceChannel.connect {
                audioProvider {
                    println(audioBuffer?.joinToString(" "))
                    println(audioBuffer?.size)
                    audioBuffer?.let { AudioFrame(it) }
                }
            }
        }
    }

    override fun onAudioData(data: ByteArray) {
        audioBuffer = data
    }
}

Links and Additionally

1. kord library: https://github.com/kordlib/kord/tree/main/voice
2. Opuc codec: https://github.com/theeasiestway/android-opus-codec
3. use audioprocessor: https://github.com/google/ExoPlayer/issues/8342

This is how I connect it to the player

@OptIn(UnstableApi::class)
class DiscordRendersFactory
    (
    context: Context,
    private val audioDataListener: AudioDataListener
) : DefaultRenderersFactory(context) {

    override fun buildAudioSink(
        context: Context,
        enableFloatOutput: Boolean,
        enableAudioTrackPlaybackParams: Boolean
    ): AudioSink {

        val defaultAudioSink = DefaultAudioSink.Builder()
            .setAudioProcessors(arrayOf(BotAudioProcessor(audioDataListener)))
            .setEnableFloatOutput(true)
            .setEnableAudioTrackPlaybackParams(enableAudioTrackPlaybackParams)
            .build()
        return defaultAudioSink
    }
}

You need to sign in to view this answers