VST Audio Processing

Topics: Audio, VST.NET Framework
Sep 23, 2013 at 6:18 PM
Hey guys, I'm back after a brief break. I have some questions about the Audio Processing class in the MidiNoteSampler namespace.

My ultimate goal is to be able to record audio in a browser from either microphone or other aux input on a computer.

I was hoping somebody could give me a brief rundown of the AudioProcessor class and the AudioBuffer that is necessary for it. Ive been reading the documentation and i am still a bit lost on how it all comes together.


Thanks!
Sep 23, 2013 at 7:45 PM
Edited Sep 23, 2013 at 7:48 PM
I'm not too sure about recording audio in a web browser. Maybe with Silverlight or ActiveX but then you won't be able to run Vst.Net in that sandbox environment.

What are you trying to do exactly?

If you want to record audio it's the Vst Host responsability to fill the Vst plugin input buffer in ProcessReplacing, think Cubase, Ableton, Sonar etc...
Sep 23, 2013 at 7:52 PM
Edited Sep 23, 2013 at 7:53 PM
Maybe you mean recording the audio out from a browser?

In that case you could try to route the browser process audio out to a Vst host but it would be much simpler to just use WASAPI instead of Vst.Net.
Sep 23, 2013 at 9:05 PM
I want to be able to record audio in the browser then apply all of the VST plugins to the audio before joining it with other tracks. So basically i want to make a recording studio that can be shared through a website. which is why I am determined to go with VST.Net as opposed to WASAPI.
Sep 24, 2013 at 2:59 AM
I'm pretty sure what you're trying to achieve is impossible. Vst.net would not be able to run properly in the sandboxed environment of a browser. Recording and real time mixing/effects without direct access to an asio driver wouldn't be fast enough. I think your only realistic option would be a Silverlight solution but these projects have turned out very underwhelming. Flash would be better suited for this kind of project but it will still be extremely limited compared to what you would achieve on a desktop. Is there a specific reason you want to run this in the browser? The security prompts for accessing the audio card, dot net and other peripherals would be much more troublesome than just downloading an application.
Sep 24, 2013 at 11:49 AM
Edited Sep 24, 2013 at 11:54 AM
I am working on a startup website that will offer musicians the possibility of collaborating on music across the internet. Our goal is to have the ability to record into the site and (then?) apply filters and eventually compile tracks together before releasing a published song.

Do you think that a plugin could be run and fed a sample through the HTML recording framework?

The main reason I am stuck on using VST is for the ability to easily apply new plugins and continue to build on the audio engine during the life cycle of the site. Can the plugins have their effects applied post recording? if so, I believe that the HTML -> VST solution might work.
Sep 24, 2013 at 3:13 PM
You would have to apply all the processing offsite in the cloud, including mixing. The GUI would also be problematic, how would you expose the plugin interface? There are good reasons why there isn't any decent DAW running in a browser. All I've seen are basic rom samplers with a mixer.

There's no such thing as an HTML recording framework, although there is a half baked specification for HTML 5:
http://www.w3.org/TR/html-media-capture/

IMO Flash is still the only realistic option if you want to record in a browser:
https://code.google.com/p/wami-recorder/
Sep 24, 2013 at 3:31 PM
That is my plan, use the cloud to process the audio. That media capture option is the one i meant, though if flash is a better option then I'll probably go with that because i have some small experience with flash. Do you think it would be possible to make a flash GUI that called on the plugins in the cloud?
Coordinator
Sep 24, 2013 at 4:55 PM
Interesting idea. If I understand it correctly you're planning to build a server-side DAW (in the cloud) and a client side gui (in the browser). I do not get how you will be able to record into the server-side DAW from the client or get the audio result output back to the client...? - I'm not a guru in html 5 ;-)

I would take the route of a 'normal' DAW host and use a plugins parameters (and programs) to generate the gui in html. So you loose all the fancy graphics a plugin might have - but you win something that will (almost) always work with plugin that support parameters.

If you do all the processing on the server take into account that scaling will be horrible. You will have to keep each session alive with a significant tax on the server resources (memory, processor).

So you're planning to use VST.NET to write the server-side DAW?
Sep 25, 2013 at 12:01 AM
You will have to clarify DAW for me, I'm still new to the whole audio studio thing. However if my guess is correct then yes this is exactly what i am trying to do. From what YuryK is saying it seems the best action would be to record the simple audio in either flash or html audio processing and then package that and send it to the server, where it can be manipulated.

I am worried about the client side now, unless i can figure out something like streaming the audio from the server it doesnt seem like the plugin effects would be able to be heard in real-time.

As for sessions, do they need to stay open perpetually? or can i save them down and then reopen when the client queries the server to edit the session again?

What kind of load will i be looking at for each session on a processor/memory?

thanks for all your help :)
Coordinator
Sep 25, 2013 at 6:44 AM
DAW=Digital Audio Workstation. Programs like Logic, CuBase, Ableton Live and Reaper are all DAWs.

Indeed, I would look for some way of streaming the audio to and from the server. I have no experience with that unfortunately.

On the server you would need some form of hosting environment (you have to write) where these sessions are managed. How to manage these session is another matter. I would think you could persist the session state when a session is inactive for a certain period of time (depends on how quick you can re-create the session later on). When the user interacts with that session again, the server will re-create it (not necessarily on the same physical server machine) and send the url to the client.

You could think of pools of instantiated plugin instances where sessions can retrieve from in order to cut down on the setup/response time. You also have to think about isolation between sessions. A crash of one session should not bring down other sessions running on the same machine. That means you probably have each session running in a separate process. You could also think of running plugins in a separate process(es) when you do not aim for real-time audio processing - that would add further robustness to the application. This decision depends on how fast you can get the audio from the client to the server and back again.

As to how many resources are required for each session? I don't know. You probably want to limit (throttle) what a session can use - otherwise it might starve other sessions of resources. To start of with you can take a real simplistic approach and limit the session to N-number of tracks using X-number of instrument plugins and Y-number of effect plugins. Something like that.

I would definitely write this one for the cloud -Windows Azure?- on which I have some experience. ;-)
Sep 25, 2013 at 11:26 AM
I have a bit of Azure experience as well, but I believe my cohorts and I are looking at HP public Cloud for the time being. Thanks for your advice, i'll definitely be back once i get into the principle coding.
Coordinator
Sep 25, 2013 at 11:59 AM
You are very welcome, I love thinking about new ideas ;-)
Sep 25, 2013 at 9:46 PM
Edited Sep 25, 2013 at 9:48 PM
If it were me, I'd just process the audio in batch in a queue like youtube does when you upload videos. You can't really encode a stream with processing in real time (decoding yes but not encoding) unless your data is sharply band limited (like speech on skype) and you need a specialized server architecture. Unless you're going for choppy music ;)

Processing in batch scales better, because you can set a finite amount of processing, say 5 plugins at a time and if you have more request they have to wait for their slots instead of being a denial of service failure.

I also think you'll need your own dedicated server, a hosting platform like azure is not made to run native code in a shared hosting environment, it's geared toward multi-session SQL database. It makes sense that a shared host wouldn't give access to run native Win32 code because of the security and performance concerns.