One of the things that I’m working on as a part of the Team Foundation implementation that I’m working on is to create a series of short (10-15 minute … for me, that’s short) videos on different topics related to working with TFS within the environment. If these were generalized TFS videos, it wouldn’t be an issue but they aren’t – they are highly customized to the specific implementation and the custom template that we’re using. As a part of it, I needed to create a virtualized environment that reproduces the production environment that I could use for the videos (as well as testing). Since I want this to be as close as possible to the actual, working, production environment, I wanted to use the same URLs as we use in production. Setting up and resolving the URLs is easy enough … it’s all a private Hyper-V environment that has its own domain controller, DNS and DHCP. Adding credentials for invisible, integrated authentication was easy as well. Or so I thought … it turned out that it wasn’t as simple as I would have thought.

The URLs were all set up. I used tfsadminutil configureconnections to set the URLs in TFS so that the registration worked correctly. I cleared the TF client cache. Yet … it still didn’t work. I got errors from the client (running on the Hyper-V host system) that basically said “ummm … so sorry, it’s not working”. Looking at the app tier’s event log, I saw a series of errors including the following (this proved to be the important one):

Detailed Message: TF53002: Unable to obtain registration data for application VersionControl.
Web Request Details
    Url: http://apptier.my.testurl.com/VersionControl/v1.0/repository.asmx [method: POST]
    User Agent: Team Foundation (devenv.exe, 9.0.30729.1)
    Headers: Content-Length=319&Content-Type=application%2fsoap%2bxml%3b+charset%3dutf-8&Accept-Encoding=gzip&Accept-Language=en-US&Expect=100-continue&Host=apptier.my.testurl.com&User-Agent=Team+Foundation+(devenv.exe%2c+9.0.30729.1)&X-TFS-Version=1.0.0.0&X-TFS-Session=09b73cf7-6f05-4528-b8d7-2b531d43a409
    Path: /VersionControl/v1.0/repository.asmx
    Local Request: False
    Host Address: 10.5.5.1
    User: TESTDOMAIN\JSawyer [authentication type: NTLM]

Exception Message: TF30063: You are not authorized to access apptier.my.testurl.com.80. (type TeamFoundationServerUnauthorizedException)

I would get this same error regardless of client (VS or TFS Web). The account being used was a domain administrator … so that ruled out that it was really an authorization issue. Besides that, it all worked find until I did all the mucking about with the URLs.

Looking about the Internets, I found a blog entry from Nick Berardi gave the solution (and yes, our current production environment in Win2K3)… I set the TFS URLs in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\DisableLoopbackCheck registry key, rebooted and voila!! It all worked!

I’ve been working on customizing a process template here. Nothing special there. I’m sure you know the process – get the template, tweak, adjust, tweak – upload to the (test) server – create a project and make sure that all looks OK. Well, while I was doing this today, I came across an mysterious error when creating the new project. Keep in mind that the template uploaded just fine … this was when selecting the template for a new project. Here’s the error:

Event Description: TF30169: The New Team Project Wizard was unable to download the process template My Process Template.

Exception Type: Microsoft.VisualStudio.Zip.ZipException

Exception Message: A file name specified within the zip file is invalid. It contains a desktop.ini file, which could be used to run authorized code.

A desktop.ini file? What on earth would that be doing in there? I suspect that this is a typo; I think they meant which could be used to run unauthorized code. Don’t get me wrong – this is a Very Good Thing. It shows that someone was thinking about the types of files in the templates and making sure that nothing harmful accidently got in there.

What happened, as you may suspect, is that there was a desktop.ini file in the folder for the process template. How it got there or why is a mystery but there it was. And, apparently, when the process template is uploaded, everything in that folder on down is also uploaded. You can see the contents of the process template as it is returned from the server in your local temp folder. If the project creation succeeds, it appears that this file is deleted so you will have to be (relatively) quick about it if you want to see it. The template will have a name like 345.tmp; a number + a .tmp extension. It’s a zip file (as the error message so nicely points out) so you can change the extension to .zip and open it up. When I did this – lo and behold – there certainly was a desktop.ini file in there. Deleting it from the template directory and re-uploading the file did, of course, solve the problem.

One lesson that I want to pass on from this – when you upload a new process template using the Process Template Manager, it uploads the entire contents of the process template folder. So make sure that you don’t have any extraneous stuff sitting in there (like desktop.ini, for example!).

This has been driving me absolutely nuts for … oh … a month or so (off and on). Here’s the scenario: I’ve got a custom task that access the Team Foundation Server to do some things before the build really gets kicking; the task is called in the BeforeEndToEndIteration target, which is one of the ones that are specified as extensible. So … nothing wrong there.

The custom task itself changed around the workspace settings for the build so that we could have 1 build for each branch of the code. Switching between the branches is done based on a property that is passed when kicking off the build (/p: switch). This allows one build definition for all of the branches, rather than having a separate build for each branch or manually changing the workspace mappings before kicking off the build. The task itself worked perfectly; the builds pulled and labeled the correct branch. Running the TFBuild standalone, debugging with MSBuild Sidekick as I previously detailed showed no problems and everything was fine. But, when running the build under full Team Build caused errors and the build itself was marked “Partially Succeeded”. Here’s what the build results looked like:

Searching for TF209001 gave me nada. Trying to figure out what was going on, I tracked the error to the Team Build logging component (in Microsoft.TeamFoundation.Build.Server.Logger.dll) and the error message was a generic one from a try{}catch{} block in the LogProjectStartedEvent. However, I couldn’t tell much else and talking with some of my friends at MSFT didn’t yield any information either.

So it bugged me. I knew that there was nothing wrong with my task … it ran just fine on its own. Even when the error happened, it did what it was supposed to do successfully and the build was actually fine. When the task was removed, everything with the logger was fine and there were no errors. But even when my task didn’t actually remap any of the workspace mappings, the logger ran into this error. It could only be some sort of mysterious interaction between the task and the TF Build logger.

Frustrated, I put this on the side burner for a while and just came back to it yesterday, determined to get to the bottom of this mystery and get everything working. Looking through everything in Reflector – something every .NET developer should have in their toolbox – didn’t show anything that really jumped out at me. The only thing that I could possibly guess was that, somehow, in the call to EnsureWorkspace(), the workspace wasn’t getting created. Checking the MSBuild properties WorkspaceName and WorkspaceOwner showed, however, that these were fine and being passed along correctly.

Baffled, I finally decided to fire up cordbg and get to the bottom of this. I knew that it was a (caught) NullReferenceException so I set the debugger to break on all NullReferenceExceptions.

(cordbg) ca e System.NullReferenceException

Next … fire up Task Manager so that I can grab the PID of the MSBuild process that Team Build used to run the build. Then kick off the build and attach to MSBuild. Finally, the debugger stopped on the exception:

First chance exception generated: (0x15e37fc) <System.NullReferenceException>
  _className=<null>
  _exceptionMethod=<null>
  _exceptionMethodString=<null>
  _message=(0x15e38e4) "Object reference not set to an instance of an object."
  _data=<null>
  _innerException=<null>
  _helpURL=<null>
  _stackTrace=(0x15e3960) <System.SByte[]>
  _stackTraceString=<null>
  _remoteStackTraceString=<null>
  _remoteStackIndex=0
  _dynamicMethods=<null>
  _HResult=-2147467261
  _source=<null>
  _xptrs=4572092
  _xcode=-1073741819
Exception is called:FIRST_CHANCE

Now we’re getting somewhere. Let’s see where we are. The where command will give us a stack trace.

(cordbg) w
Thread 0x2168 Current State:GCUnsafe spot
0)* Microsoft.TeamFoundation.Client.TeamFoundationServer::GetService +0055[native] +0014[IL] in <Unknown File Name>:<Unknown Line Number>
1)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLoggerBase::get_VersionControlServer +0055[native] +0000[IL] in <Unknown File Name>:<Unknown Lin
e Number>
2)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::EnsureWorkspace +0317[native] +0000[IL] in <Unknown File Name>:<Unknown Line Number>
3)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::LogProjectStartedEvent +0093[native] +0047[IL] in <Unknown File Name>:<Unknown Line Numb
er>
4)  Microsoft.Build.BuildEngine.EventSource::RaiseProjectStartedEvent +0061[native] +0008[IL] in <Unknown File Name>:<Unknown Line Number>
5)  Microsoft.Build.BuildEngine.EventSource::RaiseStronglyTypedEvent +0390[native] +0131[IL] in <Unknown File Name>:<Unknown Line Number>
6)  Microsoft.Build.BuildEngine.EngineLoggingServicesInProc::ProcessPostedLoggingEvents +0213[native] +0180[IL] in <Unknown File Name>:<Unknown Line N
umber>
7)  Microsoft.Build.BuildEngine.Project::Load +1608[native] +0845[IL] in <Unknown File Name>:<Unknown Line Number>
8)  Microsoft.Build.BuildEngine.SolutionWrapperProject::ScanProjectDependencies +0319[native] +0114[IL] in <Unknown File Name>:<Unknown Line Number>
9)  Microsoft.Build.BuildEngine.SolutionWrapperProject::CreateSolutionProject +0208[native] +0098[IL] in <Unknown File Name>:<Unknown Line Number>

We see the entire stack from MSBuild all the way into the logger. It turns out that the exception is happening in the GetService() method of Microsoft.TeamFoundation.Client.TeamFoundationServer. The stack trace doesn’t show use the actual source lines because we don’t have a pdb for the assembly, but it does show the IL line number, which will get us to what’s going on in ILDASM. Here’s what we find there:

IL_000f:  ldfld      class [System]System.ComponentModel.Design.ServiceContainer Microsoft.TeamFoundation.Client.TeamFoundationServer::m_serviceContainer
  IL_0014:  ldarg.1

In Reflector, this maps to:

object service = this.m_serviceContainer.GetService(serviceType);

If GetService() returns null, it’s no problem; that’s handled. But if m_serviceContainer is null … that isn’t handled and will throw an exception as soon as it tries to call GetService(). Now we’ve identified where the exception is coming from. But we still don’t know why. Investigating further, m_serviceContainer is created in the constructor of the TeamFoundationServer class, which is working just fine or we wouldn’t have even gotten this far. So how does m_serviceContainer get set to null? It turns out that it’s set to null in TeamFoundationServer::Dispose(). And only there. Hmmm … how is the TeamFoundationServer object getting disposed?

It turns out that I was disposing the TeamFoundationServer class. How did this affect the logger? Simple … both components were calling TeamFoundationServerFactory.GetServer() to return an instance of the TeamFoundationServer class. When I called this method, I wrapped the TeamFoundationServer class in a using{} block (it implements IDisposable) like a good little .NET doobie. A look into GetServer(), however, shows that it creates a cache (using a Dictionary) with a single instance of the TeamFoundationServer for each unique TFS URI. It’s  static and handed to all clients that request the same URI; this single instance lives the entire lifetime of the process and is shared around. (Considering how long it can take to initialize the session context, this isn’t a bad thing.) However, because of this, when I called Dispose(), it was on the same instance that the distributed logger was using! (BTW … this is not documented anywhere that I could find!)

The solution – somewhat counter intuitively – is to not dispose the disposable TeamFoundationServer object when you are done with it. (Don’t make a habit of it though, OK?) Once I took out the using{} block, all was well and the build ran without any hitch at all. This is only applicable to instances that are retrieved from TeamFoundationServerProxy.GetServer(); if you create the instance using new, then you should make sure that you dispose it when you are done.

This smells all kinds of wrong to me. I don’t know what the official stance would be on this, but it smells suspiciously like a bug to me. The first thing is that the TeamFoundationServer class isn’t checking to see if it is disposed before it goes about its happy, merry way, contrary to what I thought was the recommended design pattern for implementing IDisposable; this includes checking to see if the class is disposed before doing any work in a method … and throwing an ObjectDisposedException when a method is called after being disposed. If you don’t do this, then be smart enough to realize when you’ve been previous disposed (somewhere) and reinitalize when necessary (ewww … no, that’s just wrong. Don’t do that.)

So there it is, the end to the mystery.

I had this discussion with someone recently and it’s really gotten me to thinking. There is a reason that I don't use that term much to describe myself ... it is so undefined, overused and inappropriately used that, in many ways, it has lost much of its value as a title (to me at least). I cannot even begin to count how many folks I've come across that promote themselves as architects but that I certainly wouldn't call an architect. For some, it's because they don't have any technical depth but are really good at regurgitating marketing material and BS. And because they have no technical depth, they often come up with "architectures" that are very difficult to implement properly because they have no idea what it takes to actually make it happen in the real world. Unless, of course, you write marketing material. For others, they are good coders and, perhaps, could be considered a solution architect (or lead developer) ... but they don't have the vision to see outside of a narrow problem or the short term. For still others, it's a political play in the rough and tumble world of corporate politics. Finally, you have those that are super-smart and in love with building things as complex as possible, using (what they perceive to be) all the latest and greatest tools and toys. If there is a technology that they want to play with, they will make it fit into the project, regardless of whether it actually adds any real value.

Yes, there are some that I would truly consider software architects. They are a rare breed ... they have deep technical knowledge and skills; they can code in the trenches with some of the best developers. But they have something more. First, they understand deeply business priorities and can weight technology choices to meet those business priorities. It's often very hard to get developers to understand this and vocalize it; it was a challenge anytime I did an Architectural Design Session. Yes, they will look at the latest and greatest ... but only adopt it when it actually makes business sense to do so. They also understand that you need to balance priorities and have to make certain trade-offs based on those priorities - for example, sacrificing a little performance in return for higher productivity and quicker turn-around time. Or the other way; it depends on the business priorities. For example, the guys that build Dell.com have very different priorities than the guys that do internal web apps and they will (well, should) make different decisions based on those priorities. Second, they have a "Big Picture" view. They know how the different moving pieces of any even moderately complex software system (should) fit together. I remember once, at an internal Microsoft training event on software architecture, a talk by the original architect of Microsoft Transaction Server (MTS) - which has morphed into different names over the years but is still a core part of Windows. He said that architecture is all about "HST" ... "hooking shit together". In the software world, it very much is. Third, they take the long-term, practical view of development ... not just what we need to do today, but where we are going tomorrow. This is always in flux, but it is a core piece of how they look at the world. Finally, they are also pragmatic ... what can be done within the constraints that we have and with the resources that we have? They know that all things are possible, given enough time, money and resources. It may take building a new operating system or web server or middleware piece from the ground up, but it would be possible. Just not very pragmatic. Unless, of course, that happens to be your business.

I have met folks that meet the above description but there are very, very few of them. I've met far more that fit into the first paragraph. And no, I will not name any names.

In my previous post, I talked about how to debug a TFS 2008 Build using MSBuild Sidekick. But there’s more to this story … you can also do end-to-end (as in, as part of a complete TF Build) debugging of custom tasks. Regardless of the number of unit tests that you write (and they should be numerous), they don’t always show quirks in the environment that you are working in. They do, however, give you a pretty good idea of whether or not your task will work so don’t think that I’m saying that they aren’t valuable. And, of course, based on the end-to-end debugging and any bugs you may find in there, you’ll write additional unit tests to handle those scenarios.

Back to the matter at hand. It’s actually pretty easy once you have MSBuild Sidekick debugging a Team Build. Add your task into the build and call it in one of the TFBuild extensibility targets (preferably in the target that you are actually likely to use). From there, go back to Visual Studio and attach to the MSBuild Sidekick process (Debug … Attach to Process).

And once you are attached, start debugging the build with Sidekick. You’ll go straight from stepping through your targets and tasks into your code and then back again. Very nice!