VST ProcessReplacing NAudio WaveIn

Topics: Audio, Host Development, Host Processing, Newbie
May 1, 2014 at 10:30 PM
Hello VST.NET Codeplex community,

I'm currently struggling to route the Audio I get from WaveIn using NAudio (because it seems like the best way to record and buffer Line In Audio...) into a VST Plugin and then play the processed audio coming from the plugin with WaveOut.

I tried to make it as seen in this thread https://vstnet.codeplex.com/discussions/228692 but this one is using a .wav file as audio input. Also for some reason when using the VST Stream: on one computer the audio driver crashes (or something...) because it can't play any sound after i start my application and only a system reboot solves the problem and on another one i can only her crackling noises...

The most success i get when doing it this way:
WaveIn waveIn = new WaveIn();
            waveIn.BufferMilliseconds = 25;
            waveIn.DataAvailable += waveIn_DataAvailable;
            // create wave provider
            waveProvider = new BufferedWaveProvider(waveIn.WaveFormat) { DiscardOnBufferOverflow = true };
            waveOutProv = new BufferedWaveProvider(waveIn.WaveFormat) { DiscardOnBufferOverflow = true };

            // create wave output to speakers
            waveOut = new WaveOut();
            waveOut.DesiredLatency = 100;
            //waveOut.Init(waveOutProv);
            waveOut.Init(waveOutProv);

            // start recording and playback
            waveIn.StartRecording();
            waveOut.Play();

            this.sampleRate = 44100;
            this.channels = 2;
            this.blockSize = 4410;

            cont.PluginCommandStub.SetBlockSize(blockSize);
            cont.PluginCommandStub.SetSampleRate((float)sampleRate);

            vstBufManIn = new VstAudioBufferManager(channels, blockSize * channels);
            vstBufManOut = new VstAudioBufferManager(channels, blockSize * channels);

            vstBufIn = vstBufManIn.ToArray();
            vstBufOut = vstBufManOut.ToArray();

            // 4 bytes per sample (32 bit)
            naudioBuf = new byte[blockSize * channels * 4];


.....

void waveIn_DataAvailable(object sender, WaveInEventArgs e)
        {

            if (waveProvider != null)
            {
                waveProvider.AddSamples(e.Buffer, 0, e.BytesRecorded);
            }
            naudioBuf = e.Buffer;


            unsafe
            {
                fixed (byte* byteBuf = &naudioBuf[0])
                {
                    float* floatBuf = (float*)byteBuf;
                    int j = 0;
                    for (int i = 0; i < e.BytesRecorded / channels; i++)
                    {
                        vstBufIn[0][i] = *(floatBuf + j);
                        j++;
                    }
                }
            }


            cont.PluginCommandStub.ProcessReplacing(vstBufIn, vstBufOut);

            cont.PluginCommandStub.EditorIdle();

            float[] buffer = new float[e.BytesRecorded];
        
            unsafe
            {
                float* tmpBufL = ((IDirectBufferAccess32)vstBufOut[0]).Buffer;
                float* tmpBufR = ((IDirectBufferAccess32)vstBufOut[1]).Buffer;

                // multiplexing the stream requires different indexing than here
                for (int i = 0; i < (e.BytesRecorded); i++)
                {
                    buffer[i] = *(tmpBufL + i);
                }

            }
            byte[] bytebuffer = new byte[e.BytesRecorded];
            Buffer.BlockCopy(buffer, 0, bytebuffer, 0, buffer.Length);
            waveOutProv.AddSamples(bytebuffer, 0, bytebuffer.Length);


        }
This way the audio from a microphone can get through the plugin without a problem if it doesn't get changed....but the moment i add any kind of effect (or other audio processing) the sound turns into a load grizzling noise. The plugin is totally fine, btw.

Related Stackoverflow Thread: http://stackoverflow.com/questions/23172647/vst-net-route-audio

I tried many times with different code but couldn't yet find a solution to this problem. (I'm also a bit of a newbie when it comes to audio programming...)

I hope to hear from anyone.

Best Regards,
M R
Coordinator
May 2, 2014 at 7:19 AM
in the DataAvailable event handler you have this code
      fixed (byte* byteBuf = &naudioBuf[0])
      {
             float* floatBuf = (float*)byteBuf;
You interpret the data buffer provided by the wave in as floats. That is not correct. These bytes are (from what I can tell with a quick search on the NAudio codeplex site) Short ints - interleaved.

So the even shorts are Left samples and the uneven shorts are right samlples. You can use BitConverter.ToShort to do the transformation. After that you must scale the value (I assume from Int16.MinValue to Int16.MaxValue) to the [-1, 1] float range. Then you have valid VST samples.

Hope it helps,
Marc
May 3, 2014 at 5:40 PM
Thank you very much for your comment, Marc.

I changed the way i convert the input byte array to a slightly different way which also seem to work. The problem i'm still having is more about the sound coming from the plugin and converting this properly to something understandable by NAudio. It seems that if i don't do any processing inside the plugin the sound gets through the plugin and to WaveOut quite nicely. The moment when i add any processing is when it screws up, like adding an effect or looping something.

What i'm guessing is causing the problem is this part:
float[] buffer = new float[e.BytesRecorded];
        
            unsafe
            {
                float* tmpBufL = ((IDirectBufferAccess32)vstBufOut[0]).Buffer;
                float* tmpBufR = ((IDirectBufferAccess32)vstBufOut[1]).Buffer;

                // multiplexing the stream requires different indexing than here
                for (int i = 0; i < (e.BytesRecorded); i++)
                {
                    buffer[i] = *(tmpBufL + i);
                }

            }
            byte[] bytebuffer = new byte[e.BytesRecorded];
            Buffer.BlockCopy(buffer, 0, bytebuffer, 0, buffer.Length);
            waveOutProv.AddSamples(bytebuffer, 0, bytebuffer.Length);
I'm guessing that if the length of the data in the output of the plugin varies from the length of the Recorded Bytes, the sound "screws up". I couldn't yet find any solution to this. I also don't know where i could get the length of the array (float*) coming from the plugin so that i can adjust the byte array...

I'm hoping that i didn't wrote it understandable and i also hope to hear from you (or anyone really..),
M R
May 4, 2014 at 1:48 AM
Edited May 4, 2014 at 1:49 AM
I think the conversion is still wrong, here are the conversions I made for NAudio before switching to PortAudio:
[type: System.CLSCompliant(false)]
    public class VstProviderStream : WaveStream
    {
        public VstProviderStream(AudioBuffer audioBuffer)
        {
            ReadAudioBuffer = audioBuffer;
        }

        public override WaveFormat WaveFormat { get { return new WaveFormat(AudioParameters.SampleRate, 16, 1); } }
        public override long Length { get { throw new NotSupportedException(); } }
        public override long Position { get { throw new NotSupportedException(); } set { throw new NotSupportedException(); } }

        public AudioBuffer ReadAudioBuffer;

        public override int Read(byte[] buffer, int offset, int count)
        {
            ReadAudioBuffer.Process(count / 4);
            int j = 0;
            int i = 0;

            for (i = 0; i < count / 4; i++)
            {
                buffer[j] = (byte)((ReadAudioBuffer.tempBuffer[i] + 1) * (127.5));
                j++;
                buffer[j] = (byte)((short)((ReadAudioBuffer.tempBuffer[i] + 1) * (short.MaxValue / 2)) >> 8);
                j++;
            }

            return count;
        }
    }

    [type: System.CLSCompliant(false)]
    public class VstProvider16 : WaveProvider16
    {
        public VstProvider16(AudioBuffer audioBuffer)
        {
            ReadAudioBuffer = audioBuffer;
        }

        public AudioBuffer ReadAudioBuffer;

        public override int Read(short[] buffer, int offset, int sampleCount)
        {
            ReadAudioBuffer.Process(sampleCount);

            for (int i = 0; i < sampleCount; i++)
                buffer[i] = (short)((ReadAudioBuffer.tempBuffer[i] + 1) * (short.MaxValue / 2));

            return sampleCount;
        }
    }
    
    [type: System.CLSCompliant(false)]
    public class VstProvider32 : WaveProvider32
    {
        public VstProvider32(AudioBuffer audioBuffer)
        {
            ReadAudioBuffer = audioBuffer;
        }

        public AudioBuffer ReadAudioBuffer;

        public override int Read(float[] buffer, int offset, int sampleCount)
        {
            if (VstHostActions.Obj.Host == null)
            {
                for (int i = 0; i < sampleCount; i++)
                    buffer[i] = 0;

                return sampleCount;
            }

            ReadAudioBuffer.Process(sampleCount / AudioParameters.NbChannels);

            if (ReadAudioBuffer.vstBuffer != null)
            {
                int i = 0;
                int j = 0;
                for (; i < sampleCount; )
                {
                    buffer[i] = (float)ReadAudioBuffer.vstBuffer[0][j] * VstHost.velocity;
                    i++;
                    buffer[i] = (float)ReadAudioBuffer.vstBuffer[1][j] * VstHost.velocity;
                    i++;
                    j++;
                }
            }

            return sampleCount;// *ConstAudio.MaxChannels;
        }
    }
May 5, 2014 at 10:32 AM
I tried to change the way i convert the naudio buffer to the vst buffer to this:
                    for (int i = 0; i < e.BytesRecorded / 2; i++)
                    {
                        byte[] tmpbytearr = new byte[2];
                        tmpbytearr[0] = naudioBuf[i];
                        tmpbytearr[1] = naudioBuf[i + 1];
                        i++;
                        Int16 tmpint = BitConverter.ToInt16(tmpbytearr, 0);
                        
                        float f = ((tmpint / 127.5f));
                        vstBufIn[0][j] = f;
                        vstBufIn[1][j] = f;
                        j++;
                    }
he other way round looks like this:
for (int i = 0; i < (vstBufOut[0].SampleCount); i++)
                {
                    Int16 tmpint = (Int16)(vstBufOut[0][j] * 127.5);
                    j++;
                    byte[] tmparr = BitConverter.GetBytes(tmpint);
                    bytebuffer[i] = tmparr[0];
                    bytebuffer[i + 1] = tmparr[1];
                    i++;
                    tmpint = (Int16)(vstBufOut[1][j] * 127.5);
                    j++;
                    tmparr = BitConverter.GetBytes(tmpint);
                    bytebuffer[i] = tmparr[0];
                    bytebuffer[i + 1] = tmparr[1];
                    i++;
                }
The result is not grizzling, slized noise which is not resembling the input.
The input though looks like it is correct, at least judging from the volume meter inside the plugin.

I'm sorry to bother you with this probably ease problem but i just can't wrap my head around it.

Best Regards,
M R
May 5, 2014 at 4:15 PM
Edited May 5, 2014 at 4:53 PM
The sample I posted has the three Interleaved format.
It's important you pick the one matching your WaveProvider declaration in NAudio.

Edit: Removed format suggestion because I don't remember them very well.
May 5, 2014 at 4:58 PM
Edited May 5, 2014 at 5:01 PM
for (int i = 0; i < e.BytesRecorded / 2; i++)
You are converting to Short so you have two bytes per channel right?
I see a problem with divided by 2 here because you are reading the naudio samples in mono and putting them in a stereo vst buffer. hmm well maybe not, the code is hard to follow because of missing information, maybe you are trying to copy a mono channel to stereo.
The naudio samples are interleaved, for samples of short size, it's 2 bytes for the left channel followed by 2 bytes for the right.
May 5, 2014 at 5:10 PM
Edited May 5, 2014 at 5:14 PM
Regarding this one:
Int16 tmpint = BitConverter.ToInt16(tmpbytearr, 0);
float f = ((tmpint / 127.5f));
Shouldn't it be:
Int16 tmpint = BitConverter.ToInt16(tmpbytearr, 0);
float f = (tmpint / Int16.MaxValue)
And for:
Int16 tmpint = (Int16)(vstBufOut[0][j] * 127.5);
byte[] tmparr = BitConverter.GetBytes(tmpint);
bytebuffer[i] = tmparr[0];
bytebuffer[i + 1] = tmparr[1];
Shouldn't it be:
Int16 tmpint = (Int16)(vstBufOut[0][j] * Int16.MaxValue);
byte[] tmparr = BitConverter.GetBytes(tmpint);
bytebuffer[i] = tmparr[0];
bytebuffer[i + 1] = tmparr[1];
May 5, 2014 at 9:28 PM
Edited May 5, 2014 at 9:28 PM
Thank you so much for your insight, Yuri. I now changed it to this:

Input:
                    int j = 0; //Coding is at the start of the DataAvailable method of WaveIn
                    for (int i = 0; i < e.BytesRecorded; i++) 
                    {
                        byte[] tmpbytearr = new byte[2];
                        tmpbytearr[0] = naudioBuf[i]; //naudioBuf is variable holding the byte array buffered in DataAvailable (e.Buffer)
                        i++;
                        tmpbytearr[1] = naudioBuf[i];
                        i++;
                        Int16 tmpint = BitConverter.ToInt16(tmpbytearr, 0);

                        float f = ((tmpint / Int16.MaxValue));
                        vstBufIn[0][j] = f; //vstBufIn = Array which gets processed to the plugin using processreplacing

                        tmpbytearr = new byte[2];
                        tmpbytearr[0] = naudioBuf[i];
                        i++;
                        tmpbytearr[1] = naudioBuf[i];
                        i++;
                        tmpint = BitConverter.ToInt16(tmpbytearr, 0);

                        f = ((tmpint / Int16.MaxValue));
                        vstBufIn[1][j] = f;

                        j++;
                    }
Output:
                for (int i = 0; i < (vstBufOut[0].SampleCount); i++) //vstBufOut = Array holding output from Plugin
                {
                    Int16 tmpint = (Int16)(vstBufOut[0][j] * Int16.MaxValue);
                    byte[] tmparr = BitConverter.GetBytes(tmpint);
                    bytebuffer[i] = tmparr[0]; //bytebuffer is a byte array which gets later added to the WaveProvider connected to WaveOut
                    i++;
                    bytebuffer[i] = tmparr[1];
                    i++;
                    tmpint = (Int16)(vstBufOut[1][j] * Int16.MaxValue);
                    j++;
                    tmparr = BitConverter.GetBytes(tmpint);
                    bytebuffer[i] = tmparr[0];
                    i++;
                    bytebuffer[i] = tmparr[1];
                    i++;
                }
The problem being that with this code no sound gets processed to or from the plugin. Int16.MaxValue is so big that for every value the resulting float is 0.0. When i change it to 127.5 (which i took from your coding) i can hear the sliced up sound again though it sounds more like audio i put in through the microphe (though delayed and also sliced and distorded).

I feel like it's getting close to working but something seems to be still missing..

Best Regards,
M R
May 5, 2014 at 10:18 PM
The problem being that with this code no sound gets processed to or from the plugin. Int16.MaxValue is so big that for every value the resulting float is 0.0.
  • I don't think so, I'm getting good values when I test it.
The problem you're having is Dividing two integer between -1 and 1, it will round to 0.
Try changing it for:
float f = ((float)tmpint / (float)Int16.MaxValue);
May 6, 2014 at 7:49 AM
Thank you very much. That at least fixed the problem where no audio was processed. Now the audio is getting through the plugin but it's again sliced and distorded
May 6, 2014 at 9:14 AM
Double check that you are completely filling up the buffers.

Are vstBufIn[0] and vstBufIn[1] exactly the same size as e.BytesRecorded?
Is bytebuffer exactly 4 times larger than vstBufOut[0].SampleCount?
May 6, 2014 at 10:01 AM
Looking back at it, this line seems suspect:
for (int i = 0; i < (vstBufOut[0].SampleCount); i++)
Wouldn't it work better with:
for (int i = 0; i < (vstBufOut[0].SampleCount * 4); i++)
May 6, 2014 at 12:22 PM
Edited May 6, 2014 at 12:30 PM
Again thank you very much Yuri...
So i checked all of it and indeed found a few problems.
1) The bytebuffer only got filled to position 399
2) I had a not needed i plus plus in both of my for loops causing every third byte in the byte array being empty (0)
3) e.BytesRecorded is always 400 (probably because of the 25milliseconds buffer size * 16) while vstBufIn and Out was 8820 (hard coded)

That's why i changed it so that:
  • vstBufIn and Out sampleCount is 100 (e.BytesRecorded/4)
  • removed the unncessary i plus plus
  • changed the for loops appropiately so that both i variables only go to 399
Now the bytebuffer get filled completely and the sound is less distorded. Problem is: It's still quite choppy and the audio put into the microphone comes out around 5 seconds later (and then it is played like two times in a row though that might come from the choppiness). Around 2-3 seconds could come from my sound driver actually...in my final setup i will use an audio interface and not my internal soundcard...

I feel like i'm really really close to finishing this now..

PS: Regarding your second post. I did excatly that when i changed the for loops appropiately.
May 6, 2014 at 1:24 PM
2-3 seconds is extremely long, when I use Asio4all with PortAudio on a crappy SoundMax integrated sound card I probably get about 50ms of latency with block size of 512 samples. Nothing that I can detect.

Simple question, are you using headphones together with the mic to avoid feedback?

The size of your buffers are unconventional, I would go for 512 samples in and 512 samples out.
May 6, 2014 at 1:31 PM
Another question, have you tried running your program in release mode and without debugging?
May 6, 2014 at 2:25 PM
That's the thing. I'm not using Asio4All as the laptop i'm programming on will not be the same one running the final product. I'm also going to use an audio interface with that one instead of the onboard soundcard. Besides that i also think that some of the latency comes from the buffer being 25 milliseconds or even 100 i just tested with to see if the low time might cause problems. Also the WaveOut has DesiredLatency set to 100 ms which also causes latency i would think.

Is PortAudio better when it comes to latency and using it with vst.net? ... because i would switch to it if that's the case.

Yes i'm using headphones...

Should i then set the buffer time to 512/4 in that case?

I just did and i couldn't really notice much of a difference. Maybe it's a bit better.....though i can't tell. It's still choppy and the latency also still exists.
May 6, 2014 at 2:32 PM
I was thinking about setting a fixed 512 samples buffer to help you in your computations.
You seem to have difficulty transferring between buffers so choosing a fixed standard size might help.

If Vst.Net is 512 samples on 2 channels this means a total of 1024 samples.
If NAudio is 512 samples on 1 channel with 16 bit values, this means a total of 1024 bytes.
How well are your buffer defined, is NAudio mono/stereo, is it 2 bytes per samples etc.

You are converting from mono to stereo for recording and from stereo to stereo for playback and with unusual buffer size, it's not the ideal scenario if you're not experienced with audio programming. Try cutting a step in the process to isolate the problem, does routing input straight to output works etc...

Last time I checked NAudio doesn't support Asio so that's why I picked PortAudio.
May 6, 2014 at 9:51 PM
Thank you soooo very much Yuri. After your post i first decided to see if just playing the e.Buffer vom WaveIn directly at waveOut and then is noticed that i also get choppy noises and a long latency. That's why i took the code and compiled it on another computer (which will probably be the one running the final "live" version) and voilà a lot of the weird problems disappeared. Now i could actually hear the audio i put into the mic from my headphones with an acceptable latency (the remaining latency comes from my sounddriver and will hopefully be gone when i use my audio interface).

Then i thought about the mono/stereo thing and until then i thought the buffer from NAudio holds 2 channel 16 bit (2 byte) samples like this LLRRLLRR..... but the moment i changed the code inside my 2 for loops so that i go from mono NAudio to VST.NET stereo and back everything seems to work like a charm.

..I can't tell you how happy i am right now :D

This is the final code:
            WaveIn waveIn = new WaveIn();
            waveIn.BufferMilliseconds = 25;
            waveIn.DataAvailable += waveIn_DataAvailable;
            // create wave provider
            waveProvider = new BufferedWaveProvider(waveIn.WaveFormat) { DiscardOnBufferOverflow = true };
            waveOutProv = new BufferedWaveProvider(waveIn.WaveFormat) { DiscardOnBufferOverflow = true };

            // create wave output to speakers
            waveOut = new WaveOut();
            waveOut.DesiredLatency = 100;
            waveOut.Init(waveOutProv);

            // start recording and playback
            waveIn.StartRecording();
            waveOut.Play();

            vstBufManIn = new VstAudioBufferManager(2, 200);
            vstBufManOut = new VstAudioBufferManager(2, 200);

            vstBufIn = vstBufManIn.ToArray();
            vstBufOut = vstBufManOut.ToArray();
..........
naudioBuf = e.Buffer;

            unsafe
            {
                int j = 0;
                for (int i = 0; i < e.BytesRecorded; i++)
                {
                    byte[] tmpbytearr = new byte[2];
                    tmpbytearr[0] = naudioBuf[i];
                    i++;
                    tmpbytearr[1] = naudioBuf[i];
                    Int16 tmpint = BitConverter.ToInt16(tmpbytearr, 0);
                    float f = (((float)tmpint / (float)Int16.MaxValue));
                    vstBufIn[0][j] = f;
                    vstBufIn[1][j] = f;
                    j++;
                }
            }

            cont.PluginCommandStub.ProcessReplacing(vstBufIn, vstBufOut);
            cont.PluginCommandStub.EditorIdle();

            byte[] bytebuffer;
            unsafe
            {
                float* tmpBufL = ((IDirectBufferAccess32)vstBufOut[0]).Buffer;
                float* tmpBufR = ((IDirectBufferAccess32)vstBufOut[1]).Buffer;
                bytebuffer = new byte[vstBufOut[0].SampleCount * 2];
                int j = 0;
                for (int i = 0; i < (vstBufOut[0].SampleCount * 2); i++)
                {
                    Int16 tmpint = (Int16)((float)vstBufOut[1][j] * (float)Int16.MaxValue);
                    byte[] tmparr = BitConverter.GetBytes(tmpint);
                    bytebuffer[i] = tmparr[0];
                    i++;
                    bytebuffer[i] = tmparr[1];
                    tmpint = (Int16)((float)vstBufOut[1][j] * (float)Int16.MaxValue);
                    j++;
                }

            }
            waveOutProv.AddSamples(bytebuffer, 0, bytebuffer.Length);
This is probably not the nicest code but it seems to work for me :)
Coordinator
May 29, 2014 at 2:46 PM
I have uploaded an example on how one could implement sample value conversion here [release:122820].

It does splitting, interleaving and value conversions (although the sample only comes with two conversion functions to demonstrate usage).

Hope it helps,
Marc