Wednesday, 19 December 2007

The end of the year

And all of a sudden it's Christmas. In just three days I hop on a plane and fly back to Australia for the holidays, so this will most likely be the last post of 2007. 14 posts do not make a blog, so there will be plenty more to come in the new year.

I will return to the topic of Exception Shielding and I am planning a series of posts on system testing, test driven development, and integration testing as this is a hugely important and also very difficult topic. In addition to that I'll write about Commerce Server (I'm really only working on the fringes of CS at the moment so these posts may be sporadic) and SOA and any other interesting topics as they come up.

To those of you that have visited (about 20 individuals from across the globe according to FeedBurner), thank you for stopping by - and I hope you'll come by again in the next year.

Merry Christmas and a Happy New Year to all!

B for Baba Nation

It took me near on three weeks to get through all the artists on my iPod starting with "A" - and I'm afraid I've killed the challenge already. The urge to listen to music that would otherwise be months down the track (no pun intended) is just too great, and I'm struggling to find good reasons to be disciplined.

But I made it all the way through the "A"s and I started on the "B"s... but it was BackStreet Boys that killed the desire. So now I'm no longer listening in sequence. But I will try and listen to all those albums I've ignored for so long.

A time for System.Reflection

In developing a class that holds search criteria for a generic application search tool I needed to do two things: apply wildcards to the search criteria, and sanitize the search input data. As the class that holds the search criteria is basically a collection of protected or private data members and a set of public properties to expose them (and since there will not necessarily be only one single class of this type, setting up this functionality by calling each property explicitly could be really tedious - not to mention hard to maintain.

So a different measure was clearly called for. I decided to do this through reflection, and letting the class reflect upon an instance of itself and subject the relevant properties to treatment by a set of methods.

I'll show a simplified example here (don't worry, I'll leave in all the relevant bits concerning reflection). The requirements for the class, aside from holding and exposing search criteria, is that it exposes a method that you can call to sanitize the data it holds, and another method you can call to apply wildcards to the criteria.

The example class I'll use looks like this:

public class SearchCriteria
{
private string _firstName;
private string _lastName;
private string _email;

public string FirstName
{
get { return _firstName; }
set { _firstName = value; }
}

public string LastName
{
get { return _lastName; }
set { _lastName = value; }
}

public string Email
{
get { return _email; }
set { _email = value; }
}

}

It's very simple. No constructor, only three private fields and three public get/set properties. The real-life class has lots more properties but there's no need to bore you with those here - it's an example after all.

We need to both sanitize the properties and apply wildcards to them. We'll provide two public methods (Sanitize() and ApplyWildCards()) that will do the work on the properties. And this is how it looks:

using System;
using System.Reflection;

public class SearchCriteria
{
private string _firstName;
private string _lastName;
private string _email;
private delegate void ActionDelegate(PropertyInfo prop);

public string FirstName
{
get { return _firstName; }
set { _firstName = value; }
}

public string LastName
{
get { return _lastName; }
set { _lastName = value; }
}

public string Email
{
get { return _email; }
set { _email = value; }
}

public void Sanitize()
{
ModifyProperties(new ActionDelegate(Sanitize));
}

public void ApplyWildCards()
{
ModifyProperties(new ActionDelegate(ApplyWildCards));
}

private void ModifyProperties(ActionDelegate actionDelegate)
{
foreach (PropertyInfo prop in GetProperties())
{
if (prop.PropertyType == typeof(String))
actionDelegate(prop);
}
}

private PropertyInfo[] GetProperties()
{
return typeof (SearchCriteria).GetProperties();
}

private void Sanitize(PropertyInfo prop)
{
//......
}

private void ApplyWildCards(PropertyInfo prop)
{
//......
}
}

At the core of this we have the method called ModifyProperties which consumes a delegate of type ActionDelegate which is defined at the top as private delegate void ActionDelegate(PropertyInfo prop). If you look carefully you'll see that the methods Sanitize(PropertyInfo prop) and ApplyWildcards(PropertyInfo prop) both match the delegate's definition. This is in fact what allows us to pass references to these methods into ModifyProperties via Sanitize() and ApplyWildCards(). All that the ModifyProperties method does is loop through a collection of properties which is returned by the GetProperties() method and call the appropriate method (Sanitize or ApplyWildCards) via the delegate that was passed in.

I've taken out the details of Sanitize(PropertyInfo prop) and ApplyWildCards(PropertyInfo prop) because it isn't really relevant here.

Simple, isn't it? It's quite neat, I think. Now there is one thing to note here. ModifyProperties only looks at string properties, so if you have to sanitize properties of an other type you'd have to do change the code. Probably it would be wiser to create a custom attribute class and mark up the properties that can be sanitized instead of relying on property type.

Obviously this is a simple example and a first pass at creating a solution and I've not taken into consideration things such as sanitation requirements changing based on context etc., but that's fairly simple to slot in. Hopefully this has been a useful and simple real-world example of how you can use System.Reflection.

Happy coding!

Tuesday, 11 December 2007

The Endians are coming!

I was working on a piece of kit for password hashing the other day and realised that I didn't really known what the term endianism refers to. So I looked it up.

I was pleasantly surprised to find that the term, which is now an IT term more than anything, harks back to the story of Gulliver's travels.

In short, endianism refers to the order in which bytes are stored; whether the "big end" is to the left or to the right. Big-endian stores the "big-end" to the "left" (or first), and Little-endian stores the "big-end" to the right (or last). You can read a pretty good explanation of it here, and if you're interested in Jonathan Swift and how he originated the term you can read about that here.

Throw your own SqlException

Last week I was faced with a very specific problem; I needed to throw a SqlException. One of my TestFixtures required several tests in which SqlExceptions were thrown, all with different error codes (accessible via [exceptioninstance].number).

The problem was simple: you cannot instantiate a SqlException as you would an object from a class with public constructors. SqlException's construct is internal.

So what do you do? Well - my first pass at this worked for some scenario but not for others. In my tests I would create a SqlConnection instance with a bogus connection string and attempt to open it. This would throw a SqlException.

I set the "Connect Timeout" attribute of the connection string to "1" in order for Connection.Open to time out quickly.

However, this throws a specific SqlException and you really have no control over the state of the exception at all. So if you need to throw a SqlException with a specific error code you're kind of screwed.

Well, not really. This article gives an excellent example of how, using reflection, you can create your own specific SqlException. I've employed this approach with excellent results, and my tests are much faster than when I induced the exceptions with SqlConnections.

Monday, 10 December 2007

VS2005 Service Reference Wizard

I've posted a couple of times (here and here) about building WCF proxies through pre-build events. The Add Service Reference function in Visual Studio is handy for this but has some limitations (read about them in the previous posts). Generally speaking, though, the Add Service Reference function does exactly what you need, at least to get yourself up and running and your program speaking to a service.

But there's an issue. I call it a bug. And it is mighty annoying. It has wasted two afternoons of my time already. Once when I first came across it and a second time when a colleague, three weeks later, had the same problem and I could not remember what I'd done to fix the issue. Now I'm writing it down once and for all.

Using the Add Service Reference tool, if you add a URL pointing to a service's WSDL the tool will appear to find the service, but the actual generation of the proxies will fail. Why? Because the address of the WSDL is not technically the address of the service. Bloody hell - you'd think they could have fixed that issue, wouldn't you?

So make sure you remove the ?wsdl from your service reference URL.

If you're having other issues with the Add Service Reference tool that sound like there's a Content Type issue, it's likely that some of the references in your WSDL are not accessible to svcutil.exe. You can check this by adding ?xsd=xsd0 to the service reference URL (in a browser). If you get a 404 Not Found message it's likely that the account used to host your service (the one associated with the service's application pool) does not have read access to the Windows temp directory. This doesn't seem to be documented anywhere, and I found the solution on Erwyn van der Meer's blog.

Wednesday, 5 December 2007

Exception Shielding

One of the very important aspects of a robust service architecture is the concept of exception shielding. In short, exception shielding deals with the explicit boundaries between a service and its consumers (clients), and how exceptions raised within the service are handled and presented to the client. Although conceptually simple, it's not a straightforward topic when it comes to implementation. Microsoft has posted a good introduction to the concept. Shortly I'll be posting more on the topic as my own solution takes form.

A for Alanis

So, I set out to complete the iPod challenge. I don't think I'll last. Not because I have boring music on my iPod, but rather that my listening patterns really depend on my mood and what I'm doing. Today I would have preferred to listen to Roger Waters but if I keep this up I will not listen to his music until April.

So far I've listened to a-ha, Aaron Copland, Aaron Neville, ABBA, AC/DC, Ace of Base, Adrian Martinez, Aimee Mann, Air, Al Green, Al Martino, and Alanis Morissette. The song count is 282/9087.

I might break with the pattern and ensure that I listen to the artists and all their albums in alphabetical order - but I'm not sure I can do this if I cannot skip ahead to other artists that I feel like listening to.

Pre-build events, client proxies, and common libraries

Previously I posted about WCF client-proxies and how pre-build events can be used to generate these every time your client project builds.

The post sets out that the client and service share a common library of domain/business objects, and svcutil is instructed as such using the /reference flag. This approach works fine, however, you may run into problems when the shared library changes. This happened to me repeatedly during a period where I had to modify the common library which contains the classes (marked up with the [DataContract] attribute).

The problem is that the pre-build event on the client project references a running version of the WCF service. When the service interface changes, or in particular when the DataContracts that the service consume or return change, there's a discrepancy between the library referenced by the library and the library referenced by the client.

Even if you ensure that you update the client's common library code prior to the build, the pre-build event may fail. Why? Because the call to svcutil that creates the client proxy runs before the changes to the common library (in the client) are compiled.

That particular aspect of this approach isn't great. However I found that if I make the changes to the common library through the client solution and then pull these across to the service things work just fine. With SVN this is easy. The common library is a SVN external to both the client and service SVN repositories, and I can easily commit changes made through the client solution and update the service solution (I typically work on the client in one instance of Visual Studio and the service in another).

Having said all this and gone through a great deal of trouble to get everything working with pre-build events in Visual Studio 2005, it appears that the solution to this particular problem really lies with Visual Studio 2008.

Happy coding!

Thursday, 29 November 2007

Commerce Server 2007 and password hashing

Commerce Server 2007 supports hashing and encryption of database fields, such as passwords, through encryption and/or hashing algorithms that can be specified in config. Out of the box CS2007 supports three hashing algorithms, SHA1, SHA256, and MD5. I think SHA1 is the default algorithm used for one-way hashing.

If you use the UpmMembershipProvider hashing and validation is handled for you. Should you need to hash or verify a password without the help of this provider, you may be in for a fun time trying to figure out how it all works.

Microsoft has published examples of how to validate CS2007 passwords and you can deduce from the examples how the whole hashing business takes place. However, there are two curious things to note about these examples - and they're not good:

  1. The examples ignore SHA1 hashing even though that's probably what most standard setups use.
  2. The GenerateSaltValue() method is just wrong. Try running the example and passing in null as a salt value to HashPassword() and you'll see what I mean.

Before I continue, a quick overview of the standard hashing procedure is appropriate.

Given a plain-text password, we generate a random salt value that that the hash algorithm uses to hash the plain-text password. Next we turn the salt value and the plain-text passwords into byte arrays, and then combine the two byte arrays into one with the salt preceding the plain-text password. Finally we call ComputeHash(myByteArray) on our specified hash algorithm instance. This call returns another byte array which we can then traverse to generate a string. The final hashed password consists of the salt value pre-pended on the hashed password. Look at the code examples below and this will make more sense. Note also that CS2007 uses a hexadecimal representation of the bytes in our hashed password, meaning that each byte is converted to a 2-digit hexadecimal string (myByte.ToString("x2")).

Now, back to the GenerateSaltValue() method in Microsoft's example. This method creates a byte array, fills it with random integers, and turns it into a string by calling GetString() on an instance of UnicodeEncoding. In the example, the size of the byte array is 4 (specified by the SaltValueSize member), which results a 2 character string. A quick look at the HashPassword() methods shows you that any salt value passed in as a string is expected to be 8 characters. Furthermore, the byte.Parse() calls fail because the input data is of an invalid format.

What the Microsoft example has left out is that the returnes salt string should be hexadecimal. So, once the byte array has been created, each byte should be turned into a two digit hex value and appended to a string. If you do that it will work.

It took me a while to figure this out, but one particular post on Christiano's blog shed some light on the problems I was having and helped me create a better solution, shown below.

If you want to use Microsoft's example as it stands, replace the GenerateSaltValue with this method:


public string GenerateSaltValue()
{
StringBuilder builder = new StringBuilder();
UnicodeEncoding encoding = new UnicodeEncoding();
byte[] salt = new byte[SaltValueSize];
RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
rng.GetNonZeroBytes(salt);

foreach (byte outputByte in salt)
builder.Append(outputByte.ToString("x2").ToUpper());

return builder.ToString();
}


Otherwise, the below class may be a starting point for your hashing:


using System;
using System.Collections.Generic;
using System.Globalization;
using System.Security.Cryptography;
using System.Text;

public class Hasher
{
public readonly int SaltValueSize = 4;

public string Hash(string stringToHash, HashAlgorithm hash)
{
return Hash(stringToHash, null, hash);
}

public string Hash(string stringToHash, string saltValue, HashAlgorithm hash)
{
return ComputeHash(stringToHash, saltValue, hash);
}


private string ComputeHash(string stringToHash, string saltValue, HashAlgorithm hash)
{
UnicodeEncoding encoding = new UnicodeEncoding();
byte[] hashedBytes;
byte[] salt = null;
byte[] dataBuffer;
byte[] stringToHashBytes;
StringBuilder builder = new StringBuilder();

if (saltValue == null)
salt = GetSalt();
else
salt = GetSaltFromString(saltValue);

dataBuffer = new byte[encoding.GetByteCount(stringToHash) + SaltValueSize];
stringToHashBytes = encoding.GetBytes(stringToHash);

salt.CopyTo(dataBuffer, 0);
stringToHashBytes.CopyTo(dataBuffer, SaltValueSize);

hashedBytes = hash.ComputeHash(dataBuffer);

foreach (byte outputByte in salt) builder.Append(outputByte.ToString("x2").ToUpper());
foreach (byte outputByte in hashedBytes) builder.Append(outputByte.ToString("x2").ToUpper());

return builder.ToString();
}

private byte[] GetSalt()
{
byte[] saltBytes = new byte[SaltValueSize];
RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
rng.GetNonZeroBytes(saltBytes);

return saltBytes;
}

private byte[] GetSaltFromString(string saltValue)
{
byte[] saltBytes = new byte[SaltValueSize];

binarySaltValue[0] = byte.Parse(saltValue.Substring(0, 2), System.Globalization.NumberStyles.HexNumber, CultureInfo.InvariantCulture.NumberFormat);
binarySaltValue[1] = byte.Parse(saltValue.Substring(2, 2), System.Globalization.NumberStyles.HexNumber, CultureInfo.InvariantCulture.NumberFormat);
binarySaltValue[2] = byte.Parse(saltValue.Substring(4, 2), System.Globalization.NumberStyles.HexNumber, CultureInfo.InvariantCulture.NumberFormat);
binarySaltValue[3] = byte.Parse(saltValue.Substring(6, 2), System.Globalization.NumberStyles.HexNumber, CultureInfo.InvariantCulture.NumberFormat);

return saltBytes;
}

}

Wednesday, 28 November 2007

iPod challenge

What's on your iPod? How many songs, albums, and artists reside inside its shiny little chassis? If you were to ask me I would only be able to answer the question in part. The fact is, I've got way more music on my iPod than I have ever listened to. So the time has come to find out what's on my iPod.

I purchased my trusty 60GB iPod in 2006 just before moving from Australia to the UK so that I could carry my whole CD collection with me. Since then I've added a whole bunch of new CDs as well, and the result is 9087 songs, or 27.5 days of music, all stored in my pocket. But I think there are entire albums I've never listened to, or artists whose songs remain unplayed but for one or two.

So, starting tomorrow I will be playing all my iPod songs in order from 1 to 9087. How long will it take me? Well, let's see. 27.5 days of music equates to 660 hours. I can, maybe, listen to four hours of music a day. That makes for 165 days of listening. Since I mostly listen to my iPod at work that's 33 weeks. Zoiks!

Maybe by July next year I'll be done. Wow - somehow I think I might not make it through this one, but I think it would be fun to at least try because there's bound to be a few gems and a few weird hidden songs on there. So, tomorrow I start. And I'll blog about it as I go.

The blog is back

Using a custom domain name with Blogger is not as straight forward as it seems. Basically my blog went missing for a few days while http://babel-lutefisk.blogger.com, http://babel-lutefisk.net, and http://www.babel-lutefisk.net all resulted in a 404 Not Found error.

The really curious thing was that when pinging the domain names they all resolved to the same address, which is ghs.google.com. Anyway, without ranting too much about this (I am too happy to have my blog back) all credit and thanks goes to Chuck for his help.

Now I can get on with what I'm here for!

Thursday, 22 November 2007

WCF Client Proxies and Project Pre-Build events

Over the last couple of days I've had some fun setting up a new web application that speaks to a WCF web service. Both the web app (client) and the service are under development and are continually changing, and it became evident early on that updating the WCF service proxy on the client side was going to be pretty tedious.

The service proxy was originally set up using the Add Service Reference function in Visual Studio (I'm running VS2005 by the way). This is all fine and dandy, espcially since updating the proxy just requires a right-click and Update Service Reference, except it doesn't allow you to use any of the flags that the underlying svcutil tool supports via the command line. As a result the VS service reference function generates not only a service proxy but proxies for the classes that are passed between the client and service as well.

I wanted to set up something that would easily update the service proxy, omit proxy code for classes consumed or returned by the service, and allow the use of Generics in collections returned by the service. Using svcutil on the command line this can be accomplished by using the /reference and /collectionType flags. As I've already mentioned, the Add Service Reference function in VS2005 doesn't allow these flags, but you can get around this by using project pre-build events. I thought this might be useful to others out there so I've put together some instructions that may be of help.

!!Note: The code that follows will only work if you have added svcutil to your system path.

The basic setup is quite straight forward. In your solution, right-click your client project and select Properties. From the properties page select the Build Events tab. On this tab there are two text boxes in which to enter pre-build and post-build events on the command line. In the pre-build event command line box at the top, enter the following:


mkdir $(ProjectDir)SVC
chdir $(ProjectDir)SVC
svcutil http://MyServiceURL/MyService.svc /language:C# /out:MyServiceProxy.cs
copy MyServiceProxy.cs $(ProjectDir)"Service References"
chdir $(ProjectDir)
rmdir /S /Q $(ProjectDir)SVC


As you can see, this makes use of good old DOS-style commands. The first line creates a subdirectory in your project called SVC, and the second line changes the working directory to that same directory. Notice the use of the $(ProjectDir) shortcut/macro. If you click into Edit pre-build > Macros you'll see a list of the shortcuts VS makes available.

Now, the third line is what's of real interest. This calls svcutil referencing a running instance of the service (via a URL) for which I want to generate the proxy. It also specifies the output language (you can set this to VB if you like) and the filename that the proxy is written to. Then, on the last three lines the proxy class is copied into the "Service References" folder (which VS creates when you set up the Service Reference initially), and the temporary SVC folder is removed.

That's how simple it is! Now, there are two more things to accomplish, namely omitting proxy code for service return and parameter classes, and allowing the client to use Generics.

In this example the client and the service both reference a common class library that hold the DataContract classes used between services. Since this library is directly accessible to the client, it is not necessary to generate a service proxy containing DataContract proxies. To omit DataContract proxies, I simply add the /reference flag to the svcutil command like this:

/reference:$(TargetDir)MyCommonLibrary.dll

Now we're almost there. The last thing to do is to inform svcutil about the CollectionType(s) of the service. My particular service implements a couple of OperationContracts that return a List. If you don't tell svcutil this, the proxy will default to returning T[] instead. The flag to use is /collectionType (/ct is the short version), and this is how you do it:

/collectionType:System.Collections.Generic.List`1

Notice the single reverse quote before the "1" in that line, and don't confuse it with an apostrophe. By the way, the "1" signifies that Generic list has one type specified (List).

Now - there's one more thing to throw into the mix: Namespaces. You want the generated proxy to reside in the same namespace (or a related one) as your client. If you don't add the /namespace flag on svcutil the proxy generated will just contain a class and no specific namespace and that could pose a problem. But it, too, is easily fixed by adding

/namespace:*,MyNamespace

Putting all of that together the whole pre-build event looks something like this:


mkdir $(ProjectDir)SVC
chdir $(ProjectDir)SVC
svcutil http://MyServiceURL/MyService.svc /language:C# /out:MyServiceProxy.cs /reference:$(TargetDir)MyCommonLibrary.dll /collectionType:System.Collections.Generic.List`1 /namespace:*,MyNamespace
copy MyServiceProxy.cs $(ProjectDir)"Service References"
chdir $(ProjectDir)
rmdir /S /Q $(ProjectDir)SVC


Happy days! Now every time you build your client project it will look up the service and regenerate the proxy for you. That's pretty neat, I think.

A final word on the URL used to point to the running service instance. If you are sharing the client project with other developers (most likely you are) then the URL you've specified in the pre-build event needs to be accessible to the other developers as well. I get around this by setting the URL in the pre-build event to the URL for the service running on our development server, and overriding that URL in my local hosts file so that, while I develop on the client, the URL points to whereever I need it to.

Doing it this way also means that anyone can check out only the client project and it will automagically update the proxy reference against the running instance on the development server and thus pick up any changes committed by other developers working on the service!

Wednesday, 21 November 2007

Finally. Resurrection. Well, kind of..

It's 1:34am. I cannot sleep. Not for the life of me. Since my wife asked me to turn out the lights about two hours ago I've been tossing and turning. Not even listening to my iPod at the lowest volume setting worked (usually that knocks me out within minutes). So, since sleep escapes me I have decided that the time has come to get this blog going. That's why I'm sitting in the bathroom typing quietly so as not to wake my wife.

This was meant to be a resurrection of an older blog that I never got off the ground; dogspeed.blogspot.com. That blog didn't happen because I didn't really have anything to write about, and I got lazy quickly. Hopefully this one will not suffer the same fate as I believe I finally
do have something to write about; my work. Well - not the company I work for but what I do for a living. So this one's for the .Neters out there. Well, I might post other things, too - who knows?

Anyway, as I've pointed out it is very late and I should try and get some Zs. But here's to Babel Lutefisk
.net!