This is in response to a question on the StreamInsight forum where a developer was asking for a sample that uses Server.Connect.

Before I get into the details, notes:

• I used the PatternDetector sample from the StreamInsight Product Team Samples on CodePlex. Rather than write something from scratch, I just converted the existing project to VS 2010 and tweaked it.
• Add a reference to System.ServiceModel to the project.
• You need to make sure that you defined an instance for StreamInsight in the setup and installed the service. And start it. Starting it helps.
• I had to change the creds for the StreamInsight service to use my credentials instead of Network Service because I got a funky error (that I’ve never seen before) with the Event Flow Debugger. It may have something to do with the fact that I’m running on a domain controller (don’t say it … I know … but it’s just a dev machine and I need to AD for some Hyper-V VM’s that I run). I’d recommend trying to do it with Network Service but if it doesn’t work, you’ve been warned. The error is below:

Security Support Provider Interface (SSPI) authentication failed. The server may not be running in an account with identity 'DEVBIKER\J Sawyer'. If the server is running in a service account (Network Service for example), specify the account's ServicePrincipalName as the identity in the EndpointAddress for the server. If the server is running in a user account, specify the account's UserPrincipalName as the identity in the EndpointAddress for the server.

• You need to make sure that you copy the relevant DLL’s to the folder for the StreamInsight service host folder. By default, this is C:\Program Files\Microsoft StreamInsight 1.1\Host. These can be found in the \bin folder for the PatternDetector project and are:
  • StreamInsight.Samples.Adapters.SimpleTextFileReader
  • StreamInsight.Samples.Adapters.SimpleTextFileWriter
  • StreamInsight.Samples.UserExtensions.Afa
• Create a folder for the input and output folders. Actually, if you are running under your own account, this won’t be necessary. If you are running under Network Service, you will. And make sure that you give Network Service appropriate permissions.
• The User Defined Operator in the sample that I chose caused an issue when running remotely. This surprised me … usually things run pretty much unchanged. I didn’t have the time or energy to debug that so I just changed it so that it was no longer necessary.

So … the changes.

In void Main, I changed the startup code to:

if (inProc)
    using (Server server = Server.Create(streamInsightInstanceName))
    {
        RunApp(server);
    }
else
{
    using (Server server = Server.Connect(new System.ServiceModel.EndpointAddress("http://localhost/StreamInsight/" + streamInsightInstanceName)))
    {
        RunApp(server);
    }
}

As you can see, I refactored everything in the using block to a new method called “RunApp”. This made it cleaner (I thought). I suppose that I could have done it another way, but this just seemed right.

In RunApp, I changed to code to create the application to check for the existence of the application and, based on that, get or create the application object. This is below:

// Create application in the server. The application will serve 
// as a container for actual CEP objects and queries.
Console.WriteLine("Creating CEP Application");
Application application;
if (!server.Applications.ContainsKey(appName))
{
    application = server.CreateApplication(appName);
}
else
{
    application = server.Applications[appName];
}

And that is (typically) all that you need to do to change a StreamInsight app to run either in proc or remote. As I mentioned before, I did have to change the query and remove the reference to the UDO … which was very strange because I’ve never had issues with extensions before. I’m guessing that it had something to do with the implementation of the UDO.

You can download it below Program.cs (the only changed file) for the Pattern Detector sample below:

Here’s a quick UDA to do standard deviation of a window. I found it interesting that I had to take the IEnumerable<double> source and call ToArray(). If I didn’t, it would throw a NullReferenceException, although why is something of a mystery. It would be nice if I could pass in the Average from the query since that’s already calculated by the StreamInsight engine but no dice.

Note: I’ve not done any performance testing … it was copied from MSDN. Use at your own risk …

/// <summary>
/// Static class with UDA extensions for standard deviation
/// </summary>
public static class StandardDeviation
{
    /// <summary>
    /// Extension method for the UDA.
    /// </summary>
    /// <typeparam name="T">Payload type of the stream.</typeparam>
    /// <param name="window">Window to be passed to the UDA</param>
    /// <param name="map">Mapping from the payload to a float field in the payload.</param>
    /// <returns>Aggregation result.</returns>
    [CepUserDefinedAggregate(typeof(StdDevDouble))]
    public static double StdDev<T>(this CepWindow<T> window, Expression<Func<T, double>> map)
    {
        throw CepUtility.DoNotCall();
    }
}

/// <summary>
/// A UDA to calculate the standard deviation of a window
/// </summary>
public class StdDevDouble : CepAggregate<double, double>
{
    /// <summary>
    /// Computes the aggregation over a window.
    /// </summary>
    /// <param name="source">Set of events contained in the window.</param>
    /// <returns>Aggregation result.</returns>
    public override double GenerateOutput(IEnumerable<double> source)
    {
        double[] values = source.ToArray(); 

        double mean = values.AsParallel().Average();

        double standardDev = values.AsParallel().Aggregate(
            0.0,
            // do this on each thread
             (subtotal, item) => subtotal + Math.Pow((item - mean), 2),
            // aggregate results after all threads are done.
             (total, thisThread) => total + thisThread,
            // perform standard deviation calc on the aggregated result.
            (finalSum) => Math.Sqrt((finalSum / (source.Count() - 1)))
        );
        return standardDev;
    }

}

Using it in a query is simple – just make sure that you add a using statement for the namespace containing the aggregate’s definition.

var stdDeviation = from e in sourceQueryStream
             group e by e.ItemId into eachGroup
             from window in eachGroup.HoppingWindow(
                TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(1), HoppingWindowOutputPolicy.ClipToWindowEnd)
             select new {
                 ItemId = eachGroup.Key,
                 StdDev = window.StdDev(e => e.Value), 
                 Avg = window.Avg(e => e.Value), 
                 Min = window.Avg(e => e.Value), 
                 Max = window.Avg(e => e.Value), 
                 Count = window.Count()
             };

This has been driving me absolutely nuts for … oh … a month or so (off and on). Here’s the scenario: I’ve got a custom task that access the Team Foundation Server to do some things before the build really gets kicking; the task is called in the BeforeEndToEndIteration target, which is one of the ones that are specified as extensible. So … nothing wrong there.

The custom task itself changed around the workspace settings for the build so that we could have 1 build for each branch of the code. Switching between the branches is done based on a property that is passed when kicking off the build (/p: switch). This allows one build definition for all of the branches, rather than having a separate build for each branch or manually changing the workspace mappings before kicking off the build. The task itself worked perfectly; the builds pulled and labeled the correct branch. Running the TFBuild standalone, debugging with MSBuild Sidekick as I previously detailed showed no problems and everything was fine. But, when running the build under full Team Build caused errors and the build itself was marked “Partially Succeeded”. Here’s what the build results looked like:

Searching for TF209001 gave me nada. Trying to figure out what was going on, I tracked the error to the Team Build logging component (in Microsoft.TeamFoundation.Build.Server.Logger.dll) and the error message was a generic one from a try{}catch{} block in the LogProjectStartedEvent. However, I couldn’t tell much else and talking with some of my friends at MSFT didn’t yield any information either.

So it bugged me. I knew that there was nothing wrong with my task … it ran just fine on its own. Even when the error happened, it did what it was supposed to do successfully and the build was actually fine. When the task was removed, everything with the logger was fine and there were no errors. But even when my task didn’t actually remap any of the workspace mappings, the logger ran into this error. It could only be some sort of mysterious interaction between the task and the TF Build logger.

Frustrated, I put this on the side burner for a while and just came back to it yesterday, determined to get to the bottom of this mystery and get everything working. Looking through everything in Reflector – something every .NET developer should have in their toolbox – didn’t show anything that really jumped out at me. The only thing that I could possibly guess was that, somehow, in the call to EnsureWorkspace(), the workspace wasn’t getting created. Checking the MSBuild properties WorkspaceName and WorkspaceOwner showed, however, that these were fine and being passed along correctly.

Baffled, I finally decided to fire up cordbg and get to the bottom of this. I knew that it was a (caught) NullReferenceException so I set the debugger to break on all NullReferenceExceptions.

(cordbg) ca e System.NullReferenceException

Next … fire up Task Manager so that I can grab the PID of the MSBuild process that Team Build used to run the build. Then kick off the build and attach to MSBuild. Finally, the debugger stopped on the exception:

First chance exception generated: (0x15e37fc) <System.NullReferenceException>
  _className=<null>
  _exceptionMethod=<null>
  _exceptionMethodString=<null>
  _message=(0x15e38e4) "Object reference not set to an instance of an object."
  _data=<null>
  _innerException=<null>
  _helpURL=<null>
  _stackTrace=(0x15e3960) <System.SByte[]>
  _stackTraceString=<null>
  _remoteStackTraceString=<null>
  _remoteStackIndex=0
  _dynamicMethods=<null>
  _HResult=-2147467261
  _source=<null>
  _xptrs=4572092
  _xcode=-1073741819
Exception is called:FIRST_CHANCE

Now we’re getting somewhere. Let’s see where we are. The where command will give us a stack trace.

(cordbg) w
Thread 0x2168 Current State:GCUnsafe spot
0)* Microsoft.TeamFoundation.Client.TeamFoundationServer::GetService +0055[native] +0014[IL] in <Unknown File Name>:<Unknown Line Number>
1)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLoggerBase::get_VersionControlServer +0055[native] +0000[IL] in <Unknown File Name>:<Unknown Lin
e Number>
2)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::EnsureWorkspace +0317[native] +0000[IL] in <Unknown File Name>:<Unknown Line Number>
3)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::LogProjectStartedEvent +0093[native] +0047[IL] in <Unknown File Name>:<Unknown Line Numb
er>
4)  Microsoft.Build.BuildEngine.EventSource::RaiseProjectStartedEvent +0061[native] +0008[IL] in <Unknown File Name>:<Unknown Line Number>
5)  Microsoft.Build.BuildEngine.EventSource::RaiseStronglyTypedEvent +0390[native] +0131[IL] in <Unknown File Name>:<Unknown Line Number>
6)  Microsoft.Build.BuildEngine.EngineLoggingServicesInProc::ProcessPostedLoggingEvents +0213[native] +0180[IL] in <Unknown File Name>:<Unknown Line N
umber>
7)  Microsoft.Build.BuildEngine.Project::Load +1608[native] +0845[IL] in <Unknown File Name>:<Unknown Line Number>
8)  Microsoft.Build.BuildEngine.SolutionWrapperProject::ScanProjectDependencies +0319[native] +0114[IL] in <Unknown File Name>:<Unknown Line Number>
9)  Microsoft.Build.BuildEngine.SolutionWrapperProject::CreateSolutionProject +0208[native] +0098[IL] in <Unknown File Name>:<Unknown Line Number>

We see the entire stack from MSBuild all the way into the logger. It turns out that the exception is happening in the GetService() method of Microsoft.TeamFoundation.Client.TeamFoundationServer. The stack trace doesn’t show use the actual source lines because we don’t have a pdb for the assembly, but it does show the IL line number, which will get us to what’s going on in ILDASM. Here’s what we find there:

IL_000f:  ldfld      class [System]System.ComponentModel.Design.ServiceContainer Microsoft.TeamFoundation.Client.TeamFoundationServer::m_serviceContainer
  IL_0014:  ldarg.1

In Reflector, this maps to:

object service = this.m_serviceContainer.GetService(serviceType);

If GetService() returns null, it’s no problem; that’s handled. But if m_serviceContainer is null … that isn’t handled and will throw an exception as soon as it tries to call GetService(). Now we’ve identified where the exception is coming from. But we still don’t know why. Investigating further, m_serviceContainer is created in the constructor of the TeamFoundationServer class, which is working just fine or we wouldn’t have even gotten this far. So how does m_serviceContainer get set to null? It turns out that it’s set to null in TeamFoundationServer::Dispose(). And only there. Hmmm … how is the TeamFoundationServer object getting disposed?

It turns out that I was disposing the TeamFoundationServer class. How did this affect the logger? Simple … both components were calling TeamFoundationServerFactory.GetServer() to return an instance of the TeamFoundationServer class. When I called this method, I wrapped the TeamFoundationServer class in a using{} block (it implements IDisposable) like a good little .NET doobie. A look into GetServer(), however, shows that it creates a cache (using a Dictionary) with a single instance of the TeamFoundationServer for each unique TFS URI. It’s  static and handed to all clients that request the same URI; this single instance lives the entire lifetime of the process and is shared around. (Considering how long it can take to initialize the session context, this isn’t a bad thing.) However, because of this, when I called Dispose(), it was on the same instance that the distributed logger was using! (BTW … this is not documented anywhere that I could find!)

The solution – somewhat counter intuitively – is to not dispose the disposable TeamFoundationServer object when you are done with it. (Don’t make a habit of it though, OK?) Once I took out the using{} block, all was well and the build ran without any hitch at all. This is only applicable to instances that are retrieved from TeamFoundationServerProxy.GetServer(); if you create the instance using new, then you should make sure that you dispose it when you are done.

This smells all kinds of wrong to me. I don’t know what the official stance would be on this, but it smells suspiciously like a bug to me. The first thing is that the TeamFoundationServer class isn’t checking to see if it is disposed before it goes about its happy, merry way, contrary to what I thought was the recommended design pattern for implementing IDisposable; this includes checking to see if the class is disposed before doing any work in a method … and throwing an ObjectDisposedException when a method is called after being disposed. If you don’t do this, then be smart enough to realize when you’ve been previous disposed (somewhere) and reinitalize when necessary (ewww … no, that’s just wrong. Don’t do that.)

So there it is, the end to the mystery.

We’ve got the next .NET Dojos scheduled - Todd Anglin will be presenting Silverlight! There will be a dojo in Austin as well as Houston. Here are the details:

Overview:

Microsoft officially released Silverlight 2.0 in October 2008 as a platform for .NET developers to build rich internet applications (RIAs) that run cross-browser and cross-platform. Silverlight 2.0 introduces a whole new way of developing .NET applications. It is blurring the lines between what it means to develop for the web and the desktop, enabling .NET developers to rethink how they build and distribute applications. Topics covered include: Silverlight tools, Silverlight controls, Silverlight APIs, data access, and some security.

What you will learn:

Todd Anglin will guide you through a combination of lecture and hands-on labs where you will learn everything you need to know to get started with Silverlight 2.0. Specifically, you’ll learn how to work with Silverlight in Visual Studio and Expression Blend, how to use Silverlight controls, how to interact with the HTML DOM, how to access and manipulate data, how to use isolated storage, and secure your Silverlight applications. We’ll go from Silverlight 101 to 301 in about 4 hours and you’ll leave with all the essential tools you need to start building real applications with Silverlight today.

Prerequisites:

To fully participate in this workshop, attendees will need a laptop with the following:

·        Visual Studio 2008 Professional or higher (trial is okay) with Service Pack 1 installed

·        Expression Blend 2.0 or higher (trial is okay) with Service Pack 1 installed

·        Silverlight Tools for Visual Studio 2008 SP1 installed (free)

·        Deep Zoom Composer installed (free)

·        Silverlight Toolkit December 2008 or higher (available on CodePlex)

Times, dates and registration links:

Austin (Microsoft Office): March 9, 2009 1:00 – 5:00 PM. Register now.

Houston (Microsoft Office): March 13, 2009 1:00 PM – 5:00 PM. Register now.

Ever since I posted about .Net Dojo, I’ve had quite a few requests to have it elsewhere. Well, after seeing Pablo’s Day of TDD and how awesome that was, I got this little notion that it might be cool to do .Net Dojo in Austin. And … also … well, this format was kinda new and I wanted to give it a trial run or two before I started doing more. So, with that said, we’ve got the very next .Net Dojo scheduled for Austin … our first one out there! It’s going to be the same as the last Houston event and, judging from the feedback that I got on the Houston Dojo, it’s gonna rock!

Here are the details:

When: November 3, 2008  1:00 PM – 5:00 PM

Where: Microsoft Austin Office

Overview:
Windows Communication Foundation (WCF) is Microsoft’s strategic technology for building distributed, connected systems. Initially released with .NET 3.0, it has been significantly enhanced in .NET 3.5 to support the various web programming models including SOAP, POX, WS-*, REST and RSS/Atom syndications.  Topics covered include: SOA basics, WCF architecture, WCF bindings, hosting WCF services, WCF behaviors, WCF security and some design best practices.

What you will learn:
Stephen Fulcher will guide you through a combination of lecture and hands-on labs where we will learn how to implement the various web programming models with WCF. Specifically, we will cover SOA basics, WCF architecture, WCF bindings, hosting WCF services, WCF behaviors, WCF security and some design best practices.

Prerequisites:
To fully participate in this workshop, attendees will need an open mind and a laptop with:
-  Visual Studio 2008 Professional or higher (Trial is OK) with Service Pack 1
-  Windows PowerShell 1.0 (optional, but recommended)
-  Microsoft SQL Server 2005 Standard or Express
-  A CD-ROM drive to load additional components and lab files.

Now that you have the details, I know that you want to go here and register now. Hope to see y’all there!