I'd overlooked something really fundamental, however. Mike Hadlow points this out in his recent post on Code Rant. The issue is simply this: IDependencyResolver doesn't support a mechanism for releasing components, and this poses a problem when using a container such as Windsor as backing for the resolver. If you can't release components you'll eventually end up leaking memory.
In hindsight I should have stopped and listened more carefully to that gut feeling that told me I was doing something dirty when I embraced the service locator.
And for the first time I feel a bit let down by the MVC team; why did they ignore this (see the comments trail)? I have to agree with Hadlow on this: They shouldn't shout about the IDependencyResolver being container-agnostic when it blatantly isn't.
So this afternoon I'll be rolling my good old WindsorControllerFactory back into the solution.