WCF Thread Synchronization Context

There’s not much that’s more frustrating than taking the time early on in development to work out your unit tests only to discover later during integration that what worked beautifully in NUnit and VSTS tests doesn’t actually work in the context of the larger system.  I don’t think I need to spell out what’s coming next.  Yup… I’m a victim.

First, the morals of the story:  Don’t let WCF synchronization context bite you in the ass; and there’s no substitute for a good book and subsequently the knowledge you can put in your head using the right book.

Ok, here’s the scenario that you never want to find yourself in:

Ingredients:

• One console app WCF host (substitute WAS host or Windows Service host if desired)
• One windows forms app client
• One WCF service with a callback

here’s the layout:

Inside the Winforms client I get a reference to my service proxy and pass in an instance of a class that implements the service’s callback contract (which happens to be the form itself) … no big deal.  It looks something like this:

public partial class Form1 : Form, ICalculatorServiceCallback
{
    ICalculatorService proxy;
    public Form1()
    {
        InitializeComponent();
        System.ServiceModel.Channels.Binding binding =
            new NetTcpBinding(SecurityMode.Message, true);
        EndpointAddress address =
            new EndpointAddress(@"net.tcp://localhost:4000/CalculatorService");

        InstanceContext context = new InstanceContext(this);
        DuplexChannelFactory<ICalculatorService> factory =
            new DuplexChannelFactory<ICalculatorService>
            (context, binding, address);
        proxy = factory.CreateChannel();
    }

    private void Form1_Load(object sender, EventArgs e)
    {

    }

Here is the implementation of the callback contract further on down in the form:

        #region ICalculatorServiceCallback Members

        public void AddInvoked()
        {
            System.Diagnostics.Debug.WriteLine("Yo!");
        }

        #endregion

Finally, a button clicked routine (also defined in the form) to actually use the proxy:

        private void button1_Click(object sender, EventArgs e)
        {
            proxy.Add(23, 44);
        }

As for the service side, the host is a pretty typical console type host that again imperatively sets up a singleton instance of the CalculatorService which implements the ICalculatorServiceContract:

class Program
    {
        static void Main(string[] args)
        {
            ICalculatorService singletonCalculator =
                new CalculatorService();

            ServiceHost calculatorHost =
                new ServiceHost(singletonCalculator);

            NetTcpBinding binding =
                new NetTcpBinding(SecurityMode.Message, true);
            Uri address =
                new Uri(@"net.tcp://localhost:4000/CalculatorService");

            calculatorHost.AddServiceEndpoint(
                typeof(ICalculatorService), binding, address);

            calculatorHost.Open();
            Console.WriteLine(calculatorHost.State);
            Console.WriteLine
                ("CalculatorService started, press any ENTER to close...");

            Console.ReadLine();
            calculatorHost.Close();
        }
    }

The service looks pretty much like you would expect.  The important part of the actual service is that it is decorated with a ServiceBehavior attribute indicating that the InstanceContextMode shoudl be set to single, thus making our service a singleton.

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]
public class CalculatorService : ICalculatorService
{

    #region ICalculatorService Members
    private Dictionary<string, float> memory =
        new Dictionary<string, float>();

    public void Add(params float[] nums)
    {
        Console.WriteLine("The Add Method was invoked by {0}",
            System.Threading.Thread.CurrentPrincipal.Identity.Name);
        DoCallback(nums);
    }

    private void DoCallback(float[] nums)
    {
        ICalculatorServiceCallback callback =
            OperationContext.Current.GetCallbackChannel
            <ICalculatorServiceCallback>();
        callback.AddInvoked();
    }

So far nothing interesting right?  In fact I don’t even think there’s even any adding logic in there.  Oops.  Ok, here is where things start to get a little wiley.  The symptom I was seeing was manifesting itself in the Winforms client.  The CalculatorService Add() method would get called appropriately in the button event, but then the Winforms app would unexpectedly freeze.  Eventually I’d see a timeout error from the client’s proxy and then the callback would fire.  Weird.  Connectivity issues were hard to prove because the service call was being serviced, and the callback would eventually find its way to the client callback method.  The more I looked the more it became painfully obvious that I had a threading deadlock issue.  Keep in mind that this all worked in my ‘integration’ unit tests that did the hosting and the client calls in the same process.  Then I remembered reading about concurrency modes and it seemed to make sense that the default ConcurrencyMode setting on the service might be the problem.

In a nutshell, you can instruct WCF to dispatch simultaneous incoming calls to your service in several different ways with respect to threads.  This is done using the ConcurrencyMode attribute on your service.  The default mode is ConcurrencyMode.Single.

ConcurrencyMode.Single tells WCF to lock concurrent access to the service down to one thread at a time.  Incoming calls are queued up and dispatched in order.  Any call that waits in the queue longer than the service’s timeout will throw a timeout error on the client.

ConcurrencyMode.Multiple and ConcurrencyMode.Reentrant are also available.

The advantage of using ConcurrencyMode.Single (and I’m sure this is why it was chosen as the default value) is that it is threadsafe by nature.  As a service developer, you don’t have to worry about locking concurrent access to service state inside service logic because the entire service is locked down by WCF.  If you want to handle things manually you can choose ConcurrencyMode.Multiple.  ConcurrencyMode.Reentrant is a unique beast and we’ll come to that in a moment.

Now, if you plan to callback to your client(s) during the actual execution of a service operation, having the default mode of ConcurrentMode.Single just won’t do.  WCF will attempt to dispatch the callback, find that the resources it is allowed to use are being occupied by the execution of the original service method and patiently wait – DEADLOCK.  This is exactly what was happening.  Solution?  Change the ConcurrencyMode to ConcurrencyMode.Reentrant.  This gives WCF permission to abandon the original thread lock on the service and fire the callback.  Much to my surprise, this didn’t solve the problem with my windows forms app blocking.  In fact I didn’t notice any difference at all.

It was only after spending a few more hours with Juval Lowy’s excellent book : Programming WCF Services this weekend that I was able to isolate the problem to one that involved the client’s use of SynchronizationContext.

SynchronizationContext is a new beast in .Net 2.0 and affords you the ability to affinitize thread execution.  What does that mean?  Well, one example of its use is where I want to guarantee that some method executes on the same thread that the UI is running on in my windows app.  By applying the use of a synchronization context I can make this guarantee, asking .Net to marshal the thread of execution onto the thread that the UI is running on.

In WCF there are several mechanisms available to tweak the behavior of the SynchronizationContext for different aspects of your services, but the one that concerns us here is the one that affects the affinity of the thread that the client’s callback thread is dispatched on.  By default callbacks are set up to affinitize to the SynchronizationContext of the thread that creates them.  In our case, the UI thread created our callback object and execution of our callback was bound to the UI thread.  This created a problem because we were also dispatching our original call to the Add method on the UI thread.  The callback couldn’t be invoked because it was in a deadlock waiting on the original call to Add to complete.  This is why when the timeout exception was thrown, the callback finally was able to get through.

To change this behavior, you can set the UseSynchronizationContext parameter of the CallbackBehavior  attribute to false, indicating that the callback should not be affinitized to the thread that creates the callback.

The fixed portions of our failing code are below:

First change the ConcurrencyMode to Reentrant:

    [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single,
        ConcurrencyMode=ConcurrencyMode.Reentrant)]
    public class CalculatorService : ICalculatorService
    {

…

Second, change the CallbackBehavior of the object implementing our service’s callback

[CallbackBehavior(UseSynchronizationContext=false)]
    public partial class Form1 : Form, ICalculatorServiceCallback
    {
        ICalculatorService proxy;
        public Form1()
        {
            InitializeComponent();

…

I certainly hope this helps save someone some time.