This project has moved. For the latest updates, please go here.

Audio Processing (ProcessReplacing)

Topics: 64 bit, Audio, Crash, Editor UI, VST.NET Core
Oct 11, 2015 at 3:55 AM
I completed my first try at a VST, where I needed to use Core in order to import my existing code, and very happy to see it works without any problems, no crashing or such!
However, this particular VST doesn't touch the audio, just lets it pass through untouched in ProcessReplacing. (It gets VstTimeInfo in the EditorIdle sub and does some stuff to my UserControl attached to a Wpf wrapper.)

Well that's wonderful :), so feeling a little more ambitious to try a new VST idea, which actually will deal with the audio, I attempted some operations in ProcessReplacing - first a simple Pan sub that increases or reduces the sample value of one of the channels at a time, based on an event (mousemove) in a UserControl.
Also, no errors or crashes (yay!)

Then taking one more cautious step forward, I'm now trying to get VSTTimeInfo in ProcessReplacing. (just once each time).
But, here, I actually attempt to update the UserControl based on this time info, (PPqPosition) (change the display), and when I do that, I'm getting crashes if I start/stop host play more than a couple of times.
Tried to reduce the number of UserControl updates per process, but it still crashes. So I'm about ready to give up on that :-)

Maybe you just can't do that for .NET? Or because it's unmanaged without the Framework?
I have lots of commercial VSTs that update their visuals based on host time info (for example, frequency spectrum displays), so it has to be possible somehow I assume.

BTW, I'm also doing this project with Jacobi.Vst.Core, not the Framework.dll, under the assumption that you can't use a custom control with Framework. (maybe that's totally mistaken?)
Coordinator
Oct 11, 2015 at 6:24 AM
You should be able to do custom UI controls with Framework but the problem you're having will not be fixed by using it.

Keep in mind that a Host uses multiple threads for calling into your plugin. Typically the UI runs on the main (UI) thread of the host and it has a separate thread for running the audio engine - that calls into ProcessReplacing of VSTs.

So you need to synchronize access between the two. In WPF you have the Dispatcher (WinForms its Invoke) to be able to call in from another thread. That being said - you have to think carefully about how you're setting this up. The goal is to keep the ProcessReplacing code as short and fast as possible - so calling the UI from ProcessReplacing could introduce an undetermined and variable delay. It is also IMHO a dependency that goes the wrong direction - the UI calls the processing layer but not the other way around.

You need code that can queue changes from the processing layer (timing info - whatever) to be read by the UI layer to display. You still have the queue to worry about with regards to locking - lock times need to be as short as possible. To get started you could use the (newish) ConcurrentQueue<T> in .NET.

Extra: In VST 3 they have a messaging system for this purpose (which totally sucks in other areas ;-).

Hope it helps,
Marc
Oct 11, 2015 at 7:06 AM
I see, i see, thx!
(Slow learner.. gradually getting the concepts down)
It is going the wrong direction, isn't it. What i was just trying to do first is display the timeinfo in a Label in the UI. Which isn't really necessary but I wanted to view it for reference and it is that call to the UI that is crashing it.
Anyway now I just tried this, and it works without crashing:
In ProcessReplacing I set a UI-declared variable (double) to equal the PPqposition.
Calling the UI just to set that variable seems to be ok.
In the UI, i created a timer that periodically updates the label to read that variable. No crashing, yay! :)

Incidentally, in my other VST im setting the UI Label text with every call to EditorIdle, without any crashing.
Maybe that's because EditorIdle is called much much less frequently than ProcessReplacing? Surely that's the case...
Oct 11, 2015 at 7:24 AM
You can cache the timeinfo value whenever you want but I would recommend doing UI updates only on EditorIdle. There is no point on updating the UI faster than the host allows it. The other side of the coin is just as true, a host needs to spend as little time as possible in ProcessReplacing. It is generally acknowledge that ProcessReplacing is for audio processing and EdtiorIdle is for UI refresh.
Oct 11, 2015 at 7:30 AM
Edited Oct 11, 2015 at 7:31 AM
Incidentally, in my other VST im setting the UI Label text with every call to EditorIdle, without any crashing.
Maybe that's because EditorIdle is called much much less frequently than ProcessReplacing? Surely that's the case...
  • I'm guessing no. Like Marc said you need to be sure your code is thread safe and that callbacks to the UI are dispatched to the UI thread. There's no reason for a code to crash more frequently if it's called more frequently except if the code is not thread safe. Have you tried to look up the source of the crash? Is there logging facilities in your host? Do you have application errors in event viewer? If a were you I would try to narrow down this error. Perhaps it's a memory access violation.
Oct 11, 2015 at 7:42 AM
YuryK wrote:
It is generally acknowledge that ProcessReplacing is for audio processing and EdtiorIdle is for UI refresh.
Got it. Don't know why I was updating the UI in ProcessReplacing in the first place :) Ah, i remember it was because EditorIdle wasn't being called if the Host had hidden the UI window. But logically why would you want to refresh the UI if it's hidden anyway, right? All makes much more sense now, thx.
Oct 11, 2015 at 7:48 AM
Yes sure, if the host has hidden the plugin window then there isn't any good reason to update the UI, that's why you should trust the host and just update on EditorOpen and EditorIdle. If the host is defective and doesn't call EditorIdle often enough there might be a good reason why you want to update the UI on ProcessReplacing instead of EditorIdle but I don't believe this is the case for most VST host.
Oct 11, 2015 at 7:55 AM
YuryK wrote:
Incidentally, in my other VST im setting the UI Label text with every call to EditorIdle, without any crashing.
Maybe that's because EditorIdle is called much much less frequently than ProcessReplacing? Surely that's the case...
  • I'm guessing no. Like Marc said you need to be sure your code is thread safe and that callbacks to the UI are dispatched to the UI thread. There's no reason for a code to crash more frequently if it's called more frequently except if the code is not thread safe. Have you tried to look up the source of the crash? Is there logging facilities in your host? Do you have application errors in event viewer? If a were you I would try to narrow down this error. Perhaps it's a memory access violation.
At the risk of appearing unbelievably ignorant, Im not really sure how to determine if my code is "thread safe". I just run it and if it doesn't crash, great :-)

In the host Crash.txt file i found "WM_REFRESH_DETAILS - exit" just before the last entry that says "*** Fatal Exception caught." That would be the refresh i guess?
Oct 11, 2015 at 9:00 AM
Edited Oct 11, 2015 at 9:50 AM
In the host Crash.txt file i found "WM_REFRESH_DETAILS - exit" just before the last entry that says "*** Fatal Exception caught." That would be the refresh i guess?
  • You should refer to your host vendor documentation or contact your host vendor to seek out answers related to host specific error codes. Judging by the error code I believe this is a custom WIN32 message used internally by MixCraft sequencer to dispatch a call to EditorIdle but this is speculative.
At the risk of appearing unbelievably ignorant, Im not really sure how to determine if my code is "thread safe". I just run it and if it doesn't crash, great :-)
  • A first step to diagnose any errors in your plugin would be wrapping every VST .Net callbacks into your plugin (ProcessReplacing, EditorIdle, SetTimeInfo etc...) into a try catch block and logging the Exception. This would catch all managed code exception. If this doesn't work out you could also set up a last chance exception handler and set a flag in the manifest to catch unmanaged exceptions like out of memory exceptions.
Regarding thread safety... your approach of running code and crossing your fingers isn't regarded as a sound programming technique. The basic of thread safety isn't that hard to grasp. I'll try a fast crash course.

When you execute a program in Windows the OS sets up a process and a main thread for your program. A program can own many process but for simplification purpose you can view a program as equivalent for a process. A process contains at least one thread that is called the main thread. It can also contains other thread that can execute code in parallel of the main thread on processor with multiple cores. A processor core (CPU) executes machine instructions that belong to a thread. A thread can be seen the container for instruction that are to be processed by the CPU.

After setting up the process for your program the OS calls your main entry point in the context of the main thread. That means your main function is executed in the context of the main thread. This is important because without going into details only the main thread can access UI components. In .Net when an event handler is executed it is also invoked in the context of the main thread. This is useful because it means that you can update the UI in response to an event like a button click without worrying about threading issues. You can create new threads in .Net explicitly by using the thread class or implicitly by using background workers and timers.

When you are developing a VST plugin it is the VST host that manages thread callback and therefore the threading context. That means that you don't know in which threading context the host is calling you. If the host calls your plugin in the context of the main thread (UI) then you can update UI components without worry. This is often the case when the host calls EditorIdle but there is no guarantee so you shouldn't cross your fingers take it for granted and go along with it. In ProcessReplacing the host will usually call you in the context of the another thread (usually referred to as the audio thread). That means that directly updating your UI components from ProcessReplacing will crash the plugin and bring down the process (VST Host) with it because you are not updating the UI components from the main thread.

The proper solution to update UI outside of the main thread for .Net is to use Invoke with WindowsForms or dispatcher with WPF to delegate execution to the UI thread when you want to update UI components. It means that instead of executing directly and immediately the code to update the UI control in the current threading context you are pushing the code for execution in the context of the UI thread. The reason for this requirement is that the CPU might be executing different threads at different times and needs a clue from the programmer to know when it needs to pause execution of long background task and prioritize refreshing the UI to let the user feel the computer is still responsive. For example, if a VST host calls a function of your plugin you should use invoke or dispatch to update the UI. You can call invoke or dispatch with a reference to any UI control.
/// Winforms
uiControl.Invoke(() => { uiControl.Text = vstTimeInfo.ppqPosition.ToString(CultureInfo.InvariantCulture); });

// WPF
uiControl.Dispatcher.BeginInvoke(new Action(() => { uiControl.Text = vstTimeInfo.ppqPosition.ToString(CultureInfo.InvariantCulture); });
Likewise when two threads are accessing the same variable (for read/write) you need to synchronize the threads. This is known as locking. Before using the variable each thread will lock and unlock access to the shared resource. What can happen if you don't use locking is that two threads are executing instructions that work on the same piece of data at the same time. This results in undefined behavior, it's a synonym for your variable now contains corrupted data. Without locking you are running into these issues even if it seems to run just fine on your machine. This is because threading issues are caused by subtle timing errors. That's the reason why you should never just run code in a multithreaded context and assumes it works fine just because it didn't crash in your environment.

Let's assume that a VST host calls ProcessReplacing in the context of the audio thread and EditorIdle in the context of the main thread. Let's also assume that your ProcessReplacing and EditorIdle are accessing the same variable ppqPosition. To be thread safe you need to lock access to ppqPosition because it can be accessed in different threading contexts. Here's an example:
var ppqPosition;
Object lockTimeInfo = new Object();

lock (lockTimeInfo)
{
    // ppqPosition is thread safe because every thread use a lock before accessing it
    ppqPosition = vstTimeInfo.ppqPosition
}
Coordinator
Oct 11, 2015 at 9:13 AM
This might be a good time to point you towards the built in tracing in VST.NET.

VST.NET allows you to configure tracing settings for tracking specific plugins.
This has to be done in the host config file. If the host doesn't have a config file, create one (host.exe.config) and put it next to the host.exe file.

See the host sample for an example of this config.
http://vstnet.codeplex.com/SourceControl/latest#Source/Samples/Jacobi.Vst.Samples.Host/app.config

It is basically the standard .NET TraceSource/Switch config only the naming convention is VST.NET specific.

[2c]
Oct 11, 2015 at 1:38 PM
YuryK wrote:
In the host Crash.txt file i found "WM_REFRESH_DETAILS - exit" just before the last entry that says "*** Fatal Exception caught." That would be the refresh i guess?
  • You should refer to your host vendor documentation or contact your host vendor to seek out answers related to host specific error codes. Judging by the error code I believe this is a custom WIN32 message used internally by MixCraft sequencer to dispatch a call to EditorIdle but this is speculative.
Ah, how'd you know it was Mixcraft, did i mention it? :-)
Anyway, thx for the detailed and thorough explanation, it makes sense.
Actually (believe it or not :) I've been coding for quite awhile but always in .NET so i guess that's why i haven't had to worry or bother about thread-safeness. I have used Thread.Threading a couple of times in .NET.

So, now because of the interaction with non-.NET environment, this issue of thread safety comes up, right?
And I have to call Invoke as you have shown. It's very clear now how that works. Instead of calling the UI refresh directly, you Invoke it to do so later when it's in the UI thread.

(Incidentally, when I took calls to my UI out of ProcessReplacing, no more crash errors have occurred so i have no Try-Catch exceptions.)

So, essentially if i set the same variable in different subs (which may be different threads for the host), i need to lock/unlock the variables before/after in each sub.
Just getting (reading) them simultaneously is no problem, i assume...

Thanks very much!
Oct 11, 2015 at 8:51 PM
Edited Oct 11, 2015 at 8:59 PM
Ah, how'd you know it was Mixcraft, did i mention it? :-)
  • A quick google search revealed that WM_REFRESH_DETAILS is probably a WIN32 custom message in MixCraft host.
So, now because of the interaction with non-.NET environment, this issue of thread safety comes up, right?
  • There are also threading issues to be taken care of within the .Net framework itself but they are sparse indeed. For example, you can ask the .Net framework to set up a BackgroundWorker for you (it's a kind of timer with a callback function). This BackgroundWorker uses a new thread internally that isn't the main thread. The framework calls back into your user defined callback function to perform the background work in the context of the internal background worker thread. This means you'll have to use invoke or dispatcher to access UI components from that callback function because you are not in the context of the UI thread. This is different from an UI control event handler that calls back on the UI thread.
  • So it's not specific to unmanaged code. The difference in your scenario is that you are interacting with a host process that controls the threading context. When you are getting called back by a .Net function the documentation will specify in which threading context you are and you can act accordingly. However VST hosts don't guarantee a specific threading context so you should assume the worst case context and apply appropriate measures (locks, dispatcher).
So, essentially if i set the same variable in different subs (which may be different threads for the host), i need to lock/unlock the variables before/after in each sub. Just getting (reading) them simultaneously is no problem, i assume...
  • Yes this is exact, just reading is not a problem. There are techniques available to do partial locking in scenarios that don't involve simultaneous writes but the subject is definitely advanced programming because it is very tricky to get it right. In case of doubts about thread safety of a particular variable it is often suggested to simply put a full lock in place each time you read and write to it. To simplify writing thread-safe code I suggest to use as few shared variables as possible. Using static functions that do not modify variables outside the function is guaranteed to be thread safe without resorting to mitigation measures. Another technique called immutability is to never modify a variable and always create a new copy on each modification. This approach is used extensively by the .Net framework base classes like String and DateTime. Since variables are not modified using the immutability pattern it bypasses the problem of writing to variables from different threads.
Oct 16, 2015 at 10:55 PM
This is off topic, but it makes me happy to see other people discussing this stuff!
Oct 19, 2015 at 6:02 AM
YuryK,

Thanks, your explanation has been immensely helpful!
I've been practicing, and I think I've got it.
Been setting any variables having to do with the UI, always within a sub called with BeginInvoke, and have not got any errors since (except for normal coding mistakes :-)

My first and second VSTs are functioning quite nicely, very pleased :)