The Problem

.NET Aspire makes it incredibly easy to spin up local resources, including the Azure Service Bus Emulator. This is fantastic for local development as it allows us to test against a real(ish) service bus without needing an internet connection or costing a penny.

However, one thing I really missed from the “real” Azure Service Bus experience was a UI to visualise what’s going on, typically the Azure Portal or one of the various standalone apps. Being able to jump into a queue, see how many messages are sitting there, peek at the dead-letter queue, or manually send a test message to trigger a workflow is invaluable. However these tools use the ASB Admin interface, which the emulator explicitly doesn’t support.

With the emulator running in a container, you’re often left flying blind or writing throwaway console apps just to push a message onto a queue.

Luckily through the magic of Aspire we can connect to the backend SQL database and rummage around for a table which exposes the configured entities. I wasn’t able to accurately query message counts for all entity types via the database, but once we have the list of entities to parse, we can cheat and use the ordinary client to peek and get message counts that way.

The Solution: Aspire ASB Emulator UI

I decided to build a tool to fill this gap. Aspire ASB Emulator UI is basically Postman for the ASB emulator!

It’s a Blazor Interactive Server application that you can wire up directly in your Aspire AppHost. It connects to your emulator instance and gives you a nice UI to explore and interact with your entities.

Key Features

Here is what you can do with it right now:

• Entity Explorer**: View all your queues and topics. See real-time message counts for both active and dead-letter queues.
• Message Sender**: A full-featured message sender with a Monaco Editor (VS Code style). You can set broker properties like MessageId, CorrelationId, and ScheduledEnqueueTime.
• Message Viewer**: Peek or receive messages from your queues to see what’s actually going on inside them.
• Canned Messages**: Save frequently used test messages and even use AI to generate templates.
• Placeholder Syntax**: Use dynamic values in your test messages like ~newGuid~ or ~now+5m~ so you don’t have to manually edit JSON every time you send a test.

Getting Started

Getting it running is super simple. It’s designed to feel just like adding any other resource in Aspire.

First, add the hosting package to your AppHost project:

dotnet add package AspireAsbEmulatorUi.Hosting

Then, in your Program.cs, just wire it up to your Service Bus resource:

var builder = DistributedApplication.CreateBuilder(args);

// Add Azure Service Bus and configure it to run with the emulator
var serviceBus = builder
    .AddAzureServiceBus("myservicebus")
    .RunAsEmulator(c => c.WithLifetime(ContainerLifetime.Persistent));

// Add the ASB Emulator UI
builder.AddAsbEmulatorUi("asb-ui", serviceBus);

builder.Build().Run();

That’s it! When you run your Aspire project, you’ll see a new resource called “asb-ui” in the dashboard. Click the endpoint, and you’re in.

Why I Built It

I’m a big fan of great developer experience. Friction in local testing—like having to manually craft a JSON payload and run a script just to test a message handler—adds up over time. Tools like this help keep the flow going. It’s very easy to create small tools like this and integrate them with an Aspire orchestration and also to share them. Conversely it is often not very easy to deploy even simple tools like this into a real enterprise environment.

Plus, it was a great excuse to play with Blazor Interactive Server and Tailwind CSS in a .NET 10 context!

One interesting Aspire related rabbit hole…

When building the hosting extension, one challenge I faced was getting the SQL password for the emulator’s backend database. The emulator resource has an environment variable MSSQL_SA_PASSWORD, but in Aspire, these values are often expressions or references that get evaluated at runtime. To get the actual value during the resource wiring phase, you can’t just read the environment variable dictionary directly. Instead, you have to use ProcessEnvironmentVariableValuesAsync to resolve the final value.

Here is how I grabbed the password from the emulator resource to pass it to the UI:

// Process container environment variables to extract the SQL password
await sbResource.ProcessEnvironmentVariableValuesAsync(
    context.ExecutionContext,
    async (key, unprocessedValue, processedValue, exception) =>
    {
        // Capture SQL password
        if (key == "MSSQL_SA_PASSWORD")
        {
            // processedValue contains the actual resolved password!
            context.EnvironmentVariables["asb-sql-password"] = processedValue;
        }
    },
    context.Logger,
    CancellationToken.None);

Check It Out

The project is open source and available on GitHub. I’d love for you to give it a try and let me know what you think.

View on GitHub

Happy coding! 🚀

TL/DR;

Recently while preparing for a talk on integration tests, I tried a few things and discovered something I think is potentially game changing; asserting against open telemetry trace data for integration tests! This makes the tests less complex and less brittle, easier to run locally and will probably make your systems easier to support by temping you to add better telemetry earlier on in the dev loop! Basically a win-win-win-win…🧐

Background

I have always disliked integration tests because they expose just how hard it can be to run a modern distributed app locally, especially in a team scenario where people have differing preferences of tooling/setup/even OS! So we often give up on running integration tests locally altogether and end up relying on the integration tests passing in an environment… after a release… after a CI build… after we have checked in our change and hoped for the best! This is not an ideal feedback loop and could easily take 30mins+ 🥱

The other thing which annoys me about integration tests is having to assert in several ways against various different things to ensure the expected behaviour happened, E.g. using database connections to query or asb queues for assertions. Its possible of course, but think of the case of a message broker like Azure Service Bus, we need to peek at a queue to ensure first no old messages exist, then again at the right moment to assert that the expected message has appeared, then after the correct amount of time has passed we check again to see that the message has dissapeared, but we also need to check that it didn’t appear in the dead letter queue! Even after that we still aren’t certain that it was processed unless we check some other outcome like a domain event was persisted or other db update occured etc. The dance of timing here certainly plays a part in these kinds of tests being notoriously brittle. There must be a better way…

Clearly the effort required to get integration tests running locally is worthwhile and my talk set out to make that case and provide a few tips.

Enter .Net Aspire for Exceptional Local Development Experience!

.NET Aspire has been developed to solve the exact problem of running distributed apps locally and it was on my list to try anyway. Aspire allows us to define our services in terms of executable apps, services, configuration and the relationships between them all. This is also done in one single file - amazing! There are lots of excellent resources on what Aspire is and how to use it to orchestrate a distributed app, from here I will concentrate specifically how it can help with integration tests.

Open telemetry

Aspire also wires up all of the resources to send Open Telemetry data to the dashboard where it can be viewed and filtered while the app is running locally. The value of this can not be understated. Once we start using this data to see whats happening and inspect what’s gone before, it becomes addictive, we start having realisations like “if we could add that piece of data to this trace, then we’d save x minutes looking it up from 3 db queries during an incident if scenario y ever happens”. To have this kind of view of an app, at this early a stage in the development loop is amazing! See below for code to add custom trace data to a process or to bridge a process gap.

Aspire Tests

Aspire also supports testing a distributed app using the same orchestration code, meaning that when an Aspire test is run, the same dependencies get started, the same service discovery and configuration happens and we get the opportunity to create clients to our services and call APIs or send messages etc. These tests take quite a while to run due to the nature of whats actually being spun up, but they are much faster than waiting for the check-in -> CI build -> release to env -> run tests loop. Official docs here.

A Eureka moment!💡

So at this moment, while experimenting with Aspire tests, it dawned on me that all that lovely trace data would be absolutely perfect for asserting that expected behaviour ocurred as expected, its a forensic single-source-of-truth timeline of everything that happened since the app started! In other words the perfect thing to assert against!😃

Alas I also realised a few things:

There are some differences between Aspire tests the Aspire localdev ‘F5’ experience, one being that the dashboard doesn’t run during tests. The dashboard is the component that collects open telemetry data, all other orchestrated services are configured to send their telemetry data to the dashboard via gRPC. If the dashboard doesn’t run during tests, then nothing is listening to collect the trace data.☹️

Now it is possible to configure that dashboard to run during tests, but there are no public endpoints available with which to query the data. Again it is possible to use reflection to get access to the private view model, but I was unable to get that to work.😕

Asserting against trace data!🚀

So, thinking through the requirements:

• We need something that all of the resources can send data to, and given each resource runs in its own process, that likely means a new resource
• Various official OTEL backends exist, but they all seemed to be quite heavywieght for this use-case
• We need an easy way to assert against the collected trace data
• Ideally we should be able to filter just the interesting trace data, otherwise its a firehose!
• We need to be able to clear collected trace data between tests
• Ideally it should be simple to setup and in-keeping with Aspire and Aspire tests

So, after some experiementation, the simplest solution I could come up with is:

• A simple minimal API with some methods and an in-memory collection
• An implementation of OpenTelemetry.BaseExporter<Activity> which will send filtered trace data to the API, packaged up as an Aspire Client package
• A lite version of Activity to serialise and pass around
• A container image of the API in DockerHub
• An Aspire integration package to easily add the API as a container resource

Filling in some trace data gaps⛳

I noticed that in a couple of places I was missing vital info about what had happened, I was looking at this from a ‘what would be ideal to assert against’ point of view, but ‘what would be helpful when investigating an incident’ is another equally valid and useful one.

NotificationService

In my dummy Notification Service, I did not have any trace data showing me when the notification is ‘made’, because being a dummy app, it was not doing anything which is automatically traced (i.e. making an API call, sending a message or calling the database), so I added the following code:

using (var activity = Activity.StartActivity("User Notification Sent", ActivityKind.Producer))
{
    activity?.SetTag("user-notification-event.body", @event.Body);
    activity?.SetTag("user-notification-event.reference", @event.Reference);
    activity?.SetTag("user-notification-event.timestamp", @event.Timestamp.ToString("o"));

    sentNotifications.Add(@event);

    logger.LogInformation("User Notification Sent! {Reference}\nBody: {Body}\n@{Timestamp}", @event.Body, @event.Reference, @event.Timestamp);

    return;
}

Outbox

Also in my Outbox implementation, because the OutboxDispatcherHostedService runs in a separate process, when it picks up an OutboxItem to dispatch, there is no trace data from the context which originally persisted the item, so I ended up serialising the telemetry context and saving it in a json string in the database so it could be rehydrated when dispatching, this means that the span persists across both phases before the OutboxItem is persisted and then afterwards when it is dispatched.

Code to serialise the telemetry context when persisting an individual OutboxItem, found in the OutboxRepository:

private static readonly ActivitySource Activity = new(nameof(OutboxRepository));
private static readonly TextMapPropagator Propagator = Propagators.DefaultTextMapPropagator;

public async Task<long> Add(OutboxItem outboxItem, IDbTransactionWrapped? transaction = null)
{
    //... prepare update query
    using (var activity = Activity.StartActivity("Outbox Item Insertion", ActivityKind.Producer))
    {
        Dictionary<string, string> telemetryDictionary = new();
        if(activity != null)
            Propagator.Inject(new PropagationContext(activity.Context, Baggage.Current), telemetryDictionary, (carrier, key, value) => 
            {
                carrier.Add(key, value);
            });

        parameters.Add("@SerialisedTelemetry", JsonSerializer.Serialize(telemetryDictionary));

        var insertedOutboxItemId = await connection.QuerySingleOrDefault<int>(sql, parameters, transaction);

        return insertedOutboxItemId;
    }
}

Code to rehydrate just before sending an individual outbox item, found in the OutboxDispatcherhostedService:

// Hydrate the telemetry context from item.SerialisedTelemetry...
var parentContext = Propagator.Extract(default, item.SerialisedTelemetry, (serialisedTelemetry, key) => 
{
    var telemetryDictionary = JsonSerializer.Deserialize<Dictionary<string, string>>(serialisedTelemetry);

    if (telemetryDictionary == null || telemetryDictionary.Count == 0)
        return Enumerable.Empty<string>();

    if(telemetryDictionary.TryGetValue(key, out var value))
        return new List<string> { telemetryDictionary[key] };

    return Enumerable.Empty<string>();
});
Baggage.Current = parentContext.Baggage;

using var activity = Activity.StartActivity("Dispatch Message", ActivityKind.Consumer, parentContext.ActivityContext);
await messageSender.SendAsync(item.SerialisedData, item.MessagingEntityName, cancellationToken);
await outboxRepository.AddSentStatus(OutboxSentStatusUpdate.CreateSent(item.Id));
logger.LogDispatchedOutboxItem(item.Id);

Ok, how do I try it?

Setting Up QueryableTraceCollector in your Aspire project

1. Adding QueryableTraceCollector to Your Aspire Project

add the hosting nuget package to the <PackageReference Include="AJP.Aspire.Hosting.QueryableTraceCollector" Version="1.0.0" />

var queryableTraceCollectorApiKey = builder.Configuration["QueryableTraceCollectorApiKey"] ?? "123456789"; // I add a local secret in VSCode...
var queryabletracecollector = builder.AddQueryableTraceCollector("queryabletracecollector", queryableTraceCollectorApiKey)
    .WithExternalHttpEndpoints()
    .ExcludeFromManifest();

This will add a container resource which runs the API.

1. Add the client integration nuget package, probably to ServiceDefaults <PackageReference Include="AJP.Aspire.QueryableTraceCollector.Client" Version="1.0.0" />

Add

public static TBuilder ConfigureOpenTelemetry<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
{
    //... usual stuff omitted
    builder.AddOpenTelemetryExporters();

    // Filter collected trace data by DisplayName to just the bits we're interested in, these can also be wired up through config instead if you prefer...
    builder.Services.AddQueryableOtelCollectorExporter(builder.Configuration, ["Outbox Item Insertion", "User Notification Sent", "Domain Event Insertion"]); 

    return builder;
}

1. Assert against collected trace data in an Aspire test project

I do it with a nice framework🙂 see my test, the second one in here

[Test]
public void PostWeatherData_EventuallyResultsIn_AUserNotificationBeingSent2()
{
    var (given, when, then) = (new Given(), new When(), new Then());

    given.WeHaveSetupTheAppHost(out var appHost)
        .And.WeRunTheAppHost(appHost, out var app, DefaultTimeout)
        .And.WeCreateAnHttpClientForTheQueryableTraceCollector(app, appHost.Configuration, out var queryableTraceCollectorClient)
        .And.WeClearAnyCollectedTraces(queryableTraceCollectorClient)
        .And.WeCreateAnHtppClientForTheAPI(app, out var apiHttpClient, DefaultTimeout)
        .And.WeHaveSomeCollectedWeatherData(out var location, out var reference, out var requestId, out var collectedWeatherData);

    when.WeWrapTheWeatherDataInAnHttpRequest(out var httpRequest, location, reference, requestId, collectedWeatherData)
        .And.WeSendTheRequest(apiHttpClient, httpRequest, out var response);

    then.TheResponseShouldBe(response, HttpStatusCode.OK);
            
    when.WeWaitWhilePollingForTheNotificationTrace(queryableTraceCollectorClient, 9, "User Notification Sent", out var traces);
        
    then.WeAssertAgainstTheTraces(traces, traces => 
    {
        traces.AssertContainsDomainEventInsertionTag("WeatherDataCollectionInitiated");
        traces.AssertContainsDomainEventInsertionTag("SubmittedToModeling");
        traces.AssertContainsDomainEventInsertionTag("ModelUpdated");
        
        traces.AssertContainsDisplayName("Outbox Item Insertion");

        traces.AssertContains(t => t.DisplayName == "User Notification Sent"
            && t.ContainsTag("user-notification-event.body", x => x == "Dear user, your data has been submitted and included in our latest model")
            && t.ContainsTag("user-notification-event.reference", x => x == reference), 
            "Didn't find the expected user notification trace with the expected tags.");            
    });        
}

That might seem like a large test for a blog post, but consider that it is testing an entire end-to-end flow (described here in this readme) and then compare it to your integration tests and end-to-end tests at work, then it starts to look nice and terse! The fact that it does all of its assertions consistently against a single source of truth, without the need to lots of complex timing and waiting makes a previously complex problem trivially simple🤓

Bonus tip #1 - If you need to ensure that an API call is not traced

You can just create a DelegatingHandler which literally nulls the current Activity. I use this to ensure that calls to QueryableTraceCollector are not themselves traced, otherwise we end up with infinite loops, ask me how I know 🤣

public class ClientHandlerWithTracingDisabled : DelegatingHandler
{
    public ClientHandlerWithTracingDisabled(HttpMessageHandler innerHandler) : base(innerHandler){}

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Activity.Current = null;
        return await base.SendAsync(request, cancellationToken);
    }

    protected override HttpResponseMessage Send(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Activity.Current = null;
        return base.Send(request, cancellationToken);
    }
}

Bonus tip #2 - Customise display for demos

Recently I have been using VSCode for as much as possible and in particular Polyglot Notebooks for demonstrations. The Http cells are great, but for querying the QueryableTraceCollector’s API I have been using a CSharp cell where I can add an overridden ToString() method and customise the output using emojis like this:

// contents of CSharp Cell in Polyglot notebook
using System.Net.Http;
using System.Text.Json;

public class TraceData
{
    public string Resource { get; set; } = string.Empty;
    public string Source { get; set; } = string.Empty;
    public string DisplayName { get; set; } = string.Empty;
    public string TraceId { get; set; } = string.Empty;
    public string SpanId { get; set; } = string.Empty;    
    public Dictionary<string, object> Tags { get; set; } = new();

    public override string ToString()
    {
        var thirdPart = string.Empty;
        if(DisplayName == "Domain Event Insertion")
            thirdPart = Tags["domain-event.eventclassName"].ToString().Split('.').LastOrDefault();

        if(DisplayName == "User Notification Sent")
            thirdPart = $"✉️ reference: {Tags["user-notification-event.reference"].ToString()}";

        if(DisplayName == "Outbox Item Insertion")
            thirdPart = $"📫 outbox id: {Tags["outbox-item.Id"].ToString()}";
                   
        return $"{Resource.PadRight(30, '-')}> {DisplayName.PadRight(30, '-')}> {thirdPart}";
    } 
        
}

var client = new HttpClient() { BaseAddress = new Uri("http://localhost:8000"), DefaultRequestHeaders = { { "X-Api-Key", "insert-api-key-here" } } };
var response = await client.GetAsync("traces");
var responseJson = await response.Content.ReadAsStringAsync();
JsonSerializer.Deserialize<TraceData[]>(responseJson, JsonSerializerOptions.Web ).Display();

Which makes it much easier to spot things that matter.

Conclusions

Asserting against trace data with .NET Aspire and QueryableTraceCollector elevates your integration tests, not only making them easier to write but also fairly trivial to run locally and less brittle😁 By helping us to move towards something like Observability-driven-development we can gain confidence that our distributed systems behave as intended—not just at the API level, but deep within its internals. Also by making the observability more useful, earlier in the loop, we end up with more supportable systems.

All of the code from this post can be found here

Best Practices

• Isolate tests: If your QueryableTraceCollector persists between tests, be sure to call the DELETE /traces method to clear all trace data.
• Use meaningful span names: This makes querying and asserting easier.
• Assert on what matters: Focus on key operations, error handling, and context propagation.

TL/DR

We recently had a large amount of fairly procedural logic performing multiple tasks in an API request handler, we wanted a way to split it up, but maintaining a really obvious flow and ease of code navigation. This is what we came up with applied to a contrived WeatherReport API:

public async Task<OneOf<WeatherReport, Failure>> Handle(string requestedRegion, DateTime requestedDate)
{
    return await WeatherReport.Create(requestedRegion, requestedDate)
        .Then(regionValidator.ValidateRegion)
        .Then(dateChecker.CheckDate)
        .Then(CheckCache)
        .Then(weatherForecastGenerator.Generate);
}

This code is wired-up directly a minimal API, where the OneOf<WeatherReport, Failure> result is converted to an HttpResponse and returned. These are async methods, chained together with a single return path. If you like the look of that, read on!

Background

No exceptions

So in the API I mentioned earlier we are already using minimal API and the FromServices Attribute ([FromServices]IXyzRequestHandler handler) to move all processing from the API project to the Application project. This keeps the API project nice and thin.

For the return type of the Handle() method we use the excellent OneOf library, which gives us discriminated unions in C#! In this case the return type is a OneOf<Sucess, Failure> where Success just represents success and Failure is a OneOfBase representing 3 or 4 specific Failure scenarios, these get mapped to various HttpResponses in the API. This means the method signature tells us everything we need to know about all possible outcomes and we are not resorting to using Exceptions to communicate non-exceptional behaviour. For instance, bad user input is not actually exceptional, its planned for and this approach allows us to define an InvalidRequestFailure which can be returned and will then be converted into a 400 BadRequest HttpResponse.

Also because we may be interacting with external dependencies, the Handle method must be async, so the actual return type is Task<OneOf<Sucess, Failure>>

Reducing Cognitive Load

Initially, we considered using MediatR to refactor our logic, but it seemed overkill and looking through another repo where we were already using MediatR, it seemed quite hard to navigate. Each Request/Command Handler had to map its result into a new (but almost identical) Command/Request object with another Handler. So to figure out the path through the code, a fair amount of searching was nessecary.

A long while ago I did some work with Windows Workflow Foundation, which I really enjoyed. Code would be organised into functional chunks called Activities, each having In arguments, Out arguments and sometimes InOut arguments. The Activities would be arranged in a visual workflow, so the flow and ordering was really obvious. The striking thing about it, was that new team members could instantly find their way around and once inside an Activity it was immediately obvious what the scope of the task was, what inputs were available to use and what outputs would need to be set.

I wanted to find a way to replicate this declarative flow and a flavour of the functional isolation, because I want engineers to easily see what’s going on and find their way around. Our final solution shows the flow declaratively in one place and [ctrl]+[f12] takes you straight to the code in Visual Studio!

How it’s used

Here is the full version of the contrived WeatherReportRequestHandler example shown above:

public class GetWeatherReportRequestHandler : IGetWeatherReportRequestHandler
{
    private readonly IRegionValidator regionValidator;
    private readonly IDateChecker dateChecker;
    private readonly IWeatherForecastGenerator weatherForecastGenerator;

    public GetWeatherReportRequestHandler(IRegionValidator regionValidator, IDateChecker dateChecker, IWeatherForecastGenerator weatherForecastGenerator)
    {
        this.regionValidator = regionValidator;
        this.dateChecker = dateChecker;
        this.weatherForecastGenerator = weatherForecastGenerator;
    }

    public async Task<OneOf<WeatherReport, Failure>> Handle(string requestedRegion, DateTime requestedDate)
    {
        return await WeatherReport.Create(requestedRegion, requestedDate)
            .Then(report => regionValidator.ValidateRegion(report))
            .Then(report => dateChecker.CheckDate(report))
            .Then(report => CheckCache(report))
            .Then(report => weatherForecastGenerator.Generate(report));
    }

    public async Task<OneOf<WeatherReport, Failure>> CheckCache(WeatherReport report)
    {
        // Check and populate from a cache etc...
        // Methods from anywhere can be chained as long as they have the correct signature...
        return report;
    }
}

A static method named Create() returns an initialised WeatherReport object, but it actually returns a Task<OneOf<WeatherReport, Failure>> we’ll see why in a moment. The WeatherReport object contains any required state and crucially represents Success when returned.

To be chained, a method need only have the correct return type, in this case a Task<OneOf<WeatherReport, Failure>>. The chained method will probably receive a WeatherReport as an argument so it can read some state, perform its task, possibly set some resulting state and then crucially it should return the WeatherReport object if successful or a derivative of Failure if not.

Note the methods can live anywhere, they might be local or live on services injected from IoC, they just need the correct signature.

The chaining is possible because of an extension method named Then (shown below) which extends a Task<OneOf<T, TFailure>>. In the case above the T is the WeatherReport and TFailure is the Failure class mentioned above. This is why the Create() method and all of the other methods in the chain must return Task<OneOf<WeatherReport, Failure>>. The compiler infers what the Types T and TFailure are from the first method in the chain.

In this version you can see the WeatherReport variable named report being explicitly passed into each chained method, if the chained methods only have a single T argument then you can use the C# feature method group conversion to delegate, making the code even more terse and easily readable:

.Then(report => regionValidator.ValidateRegion(report))
// becomes
.Then(regionValidator.ValidateRegion)

This is shown in the TLDR section at the top.

As with any lambda expressions in C#, the func containing the chained method can capture any outer variables in scope e.g.

var validationSettings = new ValidationSettings{...}
...
.Then(report => regionValidator.ValidateRegion(report, validationSettings))

Out variables can also be used to return state from a chained method and passed into subsequent chained methods or consumed in the method containing the chain.

We have had success using a record for the T where some properties are set initially in our Create() method and other nullable properties start life as null, but are later set while creating a new version of the record using the with keyword.

How it works

Here is the simplest version of the extension method Then:

public static async Task<OneOf<T, TFailure>> Then<T, TFailure>(
  this Task<OneOf<T, TFailure>> currentJobResult,
  Func<T, Task<OneOf<T, TFailure>>> nextJob)
{
    // Inspect result of (probably already awaited) currentJobResult, if its a TFailure return it...
    var TOrFailure = await currentJobResult;
    if (TOrFailure.IsT1)
        return TOrFailure;

    // ...Otherwise do your next job
    var currentT = TOrFailure.AsT0;
    return await nextJob(currentT);
}

So the first job is to inspect the result of the previous job in the chain, we await it, but are pretty sure that its already been awaited and therefore the Result is already available. Note this is fine because we are using the full blown Task, if it were ValueTask we would not be able to await a second time. The OneOf library has methods for us to determine if if the Result contains the T (WeatherReport in this case) or TFailure (a Failure in this case). IsT0() will return true if the object contains the first possiblity in the discriminated union, IsT1() will return true for the second item etc.

If the result contains a TFailure from earlier in the chain we immediately return that Failure without performing our Task, all of the following Then's in the chain do the same. This means the first Failure returned by a chained task is returned and no more chained tasks are executed.

Otherwise if the Result was successful, which means it contained a T (WeatherReport in this case) then we should call and await the Func nextJob passing in the T (WeatherReport in this case). This is how the chain is continued.

What if something goes wrong?

We had a case where we were creating two transactions, one after the other in the chain, but if the first succeeded and the second failed, we needed a way to cancel the first transaction. Initially this code was hidden in the second transaction creation method, but a team member pointed out that was counter-intuitive. To solve this problem a second overload of the Then method is provided which takes a second Func called onFailure as an argument. We inspect the Result of nextJob in the same way and if it fails, we invoke and await onFailure before returning the TFailure as before.

Here is the overload of the extension method Then with onFailure:

public static async Task<OneOf<T, TFailure>> Then<T, TFailure>(
  this Task<OneOf<T, TFailure>> currentJobResult,
  Func<T, Task<OneOf<T, TFailure>>> nextJob,
  Func<T, TFailure, Task<TFailure>> onFailure)
{
    // Inspect result of (probably already awaited) currentJobResult, if its a TFailure return it...
        var TOrFailure = await currentJobResult;
        if (TOrFailure.IsT1)
            return TOrFailure;

        // ...Otherwise do your next job ... 
        var currentT = TOrFailure.AsT0;
        var result = await nextJob(currentT);

        // If next job returned a T (Success) return it...
        if (result.IsT0)
            return result;        
        
        // Otherwise invoke onFailure and return the final resulting TFailure...
        var finalFailure = await onFailure(currentT, result.AsT1);
        return finalFailure;
}

It can be used like this:

return await PaymentDetails.Create(request)
        .Then(d => validator.ValidateRequest(d))
        .Then(d => paymentRepository.Save(d))
        .Then(d => transactionService.CreateDebitTransaction(d))
        .Then(d => transactionService.CreateCreditTransaction(d), onFailure: d, f => transactionService.CancelDebitTransaction(d, f))
        .Then(d => eventDispatcher.Dispatch(d));

We also have a method named IfThen() which takes an additional func<T, bool> named condition so that the nextJob will only be invoked if condition evaluates to True; I’m sure there are other variations which will be added.

Conclusion

So if you also have multiple tasks to perform off the back of an API request or when handling a Service Bus message etc, maybe consider this method of chaining methods together. Maintain a single return path, place the methods wherever they best fit and make life easy for your team by expressing declaratively the flow of tasks with super-easy code navigation.

This code is avilable in a Nuget package and in this GitHib repository, but its basically a few small extension methods, so copy/paste away!

Hope you found this interesting, thanks for reading :)

TL/DR

This post introduces a method of organising test code into shared, reusable chunks, enabling complex tests to be expressed in a terse english-like fashion, which looks like this:

[Fact]
public void Return_badRequest_when_request_is_invalid()
{
    Given.UsingThe(apiWebApplicationFactory)
        .TheTransactionsApiCreateEndpointWillReturn(HttpStatusCode.OK)
        .And.WeHaveAnInvalidPaymentRequest(out var invalidPaymentRequest, currency)
        .And.TheServerIsStarted();

    When.UsingThe(apiWebApplicationFactory)
        .WeWrapThePaymentRequestInHttpRequestMessage(invalidPaymentRequest, out var httpRequestMessage)
        .And.WeSendTheMessageToTheApi(httpRequestMessage, out var response);
    
    using (new AssertionScope())
    {
        Then.UsingThe(apiWebApplicationFactory)
            .TheResponseCodeShouldBe(response, HttpStatusCode.BadRequest)
            .And.TheBodyShouldNotBeEmpty(response, out var body)
            .And.TheBodyShouldContainAProblemDetails(body, details => details.Errors.Count.Should().Be(1))
            .And.APaymentRecordWasNotSaved()
            .And.NoPaymentStatusRecordsWereSaved();
    }
}

Hopefully you agree that while still outlining precisely what is being set up, executed and asserted against, this test is easy to understand. In fact you could probably show it to your Product Owner without requiring too much explanation.

Background

I have alluded to this pattern in previous blog posts, but I’ve been using it for a while now on many different projects, and it continues to be extremely useful so I thought it deserved a blog in its own right.

In our team we favour using Component tests to create the majority of our code coverage, filling in with Unit tests where necessary. Then we add minimal Integration tests to prove out integrations with external services and finally end-to-end tests to test flows from start to finish. I guess this roughly aligns to the testing trophy idea as opposed to the testing triangle.

Our Component tests are like internal Integration tests, we we aim to exercise as much of the app as possible, just mocking the external dependencies at the outermost point. So, for example, we use WebApplicationFactory to test from the Program onwards, setting up Mocks and switching them in place of external service interfaces, such as database or service bus connections. The rest of the real services/components under test obviously do not know that the external dependencies are mocked.

This kind of test provides lots of value, including a relatively realistic local debugging experience and finding issues that traditional unit testing would not be expected to find. For instance, if an injected service was not registered in the IoC container or a required config value was missing, these tests would not pass.

There are downsides to this approach; there is a fair amount of code to initially set everything up and this in turn makes it hard to do TDD. The tests themselves can also become quite large unless you have a strategy for organising and sharing code across tests…

Given, When and Then classes

In order to solve some of those problems, we separate out the arrange, act and assert parts of a test into Given, When and Then classes, every method on each of these classes returns itself, which results in a nice fluent interface making the tests shorter and improving readability. This approach encourages code sharing and consistency across tests.

The Given class

is responsible for the ‘arrange’ part of the test. It contains methods for setting up behaviour on the Mocks, creating payloads and/or other bits of test data. When setting up Mocked behaviours, I like to name methods using the future tense e.g. TheTransactionsApiWillReturnTheStatusCode(HttpStatusCode.BadRequest)

The When class

is responsible for the ‘act’ part of the test. It contains methods for kicking off the process being tested, so sending a request to an API, sending a service bus message, inserting data into a database or creating events in an event store etc.

The Then class

is responsible for the ‘assert’ part of the test. It contains methods for making assertions. One useful trick is to have a method which takes an Action to allow the tests to do their own assertions with any assertion library, for example you might have a method named TheHttpResponseContainsABody(HttpResponseMessage response, Action<string> assertAgainstBody) this can then be used with FluentAssertions like this: And.TheHttpResponseContainsABody(response, body => body.Should().Be("Alive!"))

In the above example, the whole Then section is wrapped in a using (new AssertionScope()) this means that all assertions will be evaluated before any failed assertions cause the test to fail, so all of the failed assertions will be reported, rather than just the first one.

Shared stuff

All three classes share an instance of whatever owns the Mocks and useful properties or fields, let’s call this the context. Each class has a static method UsingThe() which assigns this shared state object and returns a new Given, When or Then. Each class also has a dummy And property which just returns itself meaning the tests read more like English sentences.

Out variables!

So no-one likes out variables, myself included, but here they offer a way of defining state inline without breaking the flow/readability of the tests, which I think really works well. It also makes the variables easy to trace through the test steps.

Some code

Simplified example Given, When and Then classes are included below to get you going. These classes are necessarily specific to the app under test, so we find we start from scratch each time, here is a handy gist containing the bare bones classes ready to add methods to.

Here is an example of a Given class with methods for setting up testing an API:

public class Given
{
    private readonly WeatherReportApiWebApplicationFactory apiWebApplicationFactory;

    public Given(WeatherReportApiWebApplicationFactory apiWebApplicationFactory)
    {
        this.apiWebApplicationFactory = apiWebApplicationFactory;
        apiWebApplicationFactory.MockWeatherReportFetcher.Invocations.Clear();
    }

    public static Given UsingThe(WeatherReportApiWebApplicationFactory apiWebApplicationFactory) => new(apiWebApplicationFactory);
    public Given And => this;
    
    public Given TheWeatherReportFetcherApiEndpointWillReturn(HttpStatusCode statusCode, Refit.ProblemDetails? problemDetails = null)
    {
        apiWebApplicationFactory.MockWeatherReportFetcher
            .SetupRequest(HttpMethod.Get, r => r.RequestUri == someUri)
            .ReturnsResponse(statusCode, new StringContent(JsonSerializer.Serialize(problemDetails), Encoding.UTF8));

        return this;
    }    

    public Given WeHaveAWeatherReportRequestPayload(out WeatherForecastRequest requestPayload, string currency)
    {
        paymentRequest = new WeatherForecastRequest(
            // populate request here...
        );

        return this;
    }    

    public Given TheServerIsStarted()
    {
        apiWebApplicationFactory.Start();
        return this;
    }
}

Here is an example of a When class with methods for testing an API:

public class When
{
    private readonly WeatherReportApiWebApplicationFactory apiWebApplicationFactory;

    public When(WeatherReportApiWebApplicationFactory apiWebApplicationFactory)
    {
        this.apiWebApplicationFactory = apiWebApplicationFactory;
    }

    public static When UsingThe(WeatherReportApiWebApplicationFactory apiWebApplicationFactory) => new(apiWebApplicationFactory);
    public When And => this;

    public When WeWrapTheRequestInHttpRequestMessage(WeatherForecastRequest requestPayload, out HttpRequestMessage httpRequest)
    {        
        httpRequest = new HttpRequestMessage(HttpMethod.Post, DomesticOrchestratorApiWebApplicationFactory.DomesticPaymentUri);
        httpRequest.Headers.Add("AdditionalRequiredHeaderName", JsonSerializer.Serialize(someValue));
        httpRequest.Content = new StringContent(JsonSerializer.Serialize(requestPayload), Encoding.UTF8, "application/json");

        return this;
    }

    public When WeSendTheMessageToTheApi(HttpRequestMessage httpRequest, out HttpResponseMessage response)
    {
        response = apiWebApplicationFactory.HttpClient.SendAsync(httpRequest).GetAwaiter().GetResult();
        return this;
    }
}

Async methods

The methods on these classes are necessarily all ordinary synchronous methods, otherwise the await keyword would prevent the method chaining. So to execute some asynchronous process like calling an API we do use .GetAwaiter().GetResult(); The reason I’m Ok with this is that the test code being synchronous actually simplifies things. If the test code and the code-under-test were both async, I think there would be more chance of race conditions and more complex code required to prevent them. This way, if we need to wait for an async process to complete, e.g. for a service bus handler to finish handling a message, we can add a method Then... And.WeWaitSomeTimeForProcessingToComplete(timeToWaitInMilliSeconds: 500)

Here is an example of a Then class with methods for testing an API:

public class Then
{
    private readonly WeatherReportApiWebApplicationFactory apiWebApplicationFactory;

    public Then(WeatherReportApiWebApplicationFactory apiWebApplicationFactory)
    {
        this.apiWebApplicationFactory = apiWebApplicationFactory;
    }

    public static Then UsingThe(WeatherReportApiWebApplicationFactory apiWebApplicationFactory) => new(apiWebApplicationFactory);
    public Then And => this;

    public Then TheResponseCodeShouldBe(HttpResponseMessage response, HttpStatusCode code)
    {
        response.Should().NotBeNull();
        response.StatusCode.Should().Be(code);
        return this;
    }

    public Then TheBodyShouldNotBeEmpty(HttpResponseMessage response, Action<string> assertAgainstBody)
    {
        var body = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
        assertAgainstBody(body);

        return this;
    }
}

In conclusion

So there it is, a simple strategy for organising your test code with, so lightweight it doesn’t even constitute a framework! Hope you find it helpful and thanks for taking the time to read :)

Background

At ClearBank, we are currently preferring Dapper to Entity Framework, this means we are writing more stored procedures and naturally want to cover those stored procedures with tests. At first, we just used xUnit tests, which worked perfectly well but the setup and teardown of test data proved a little cumbersome. We then tried switching approach to the tSQLt framework, which again worked perfectly well and had the added advantage of running against a temporary database instance with a DacPac project deployed and running each test in its own SQL transaction. However, as developers spoilt by the lovely syntax of modern languages like C#, we found the tSQLt tests quite unpleasant to both read and write.

I thought there must be a better way to combine the best bits of both worlds and this SQL test framework is what I came up with. Hopefully, it will be useful to other people too.

In a nutshell

The framework allows tests to be written in C# using familiar test frameworks (xUnit is used in the examples). The tests can be run against localDb or some other SQL server. A DacPac can optionally be deployed. Each test is isolated from the other tests. Given, When and Then helper classes are included to and also a way to define tabular data for test data setup and/or assertions. More on those things later. We are also using this package to test repository classes in addition to stored procedures.

A test looks like this:

[Fact]
public void spFetchOrderById_returns_an_order_matching_the_supplied_order_Id()
{
    // the numbers in the comments relate to the explanation below: 
    new LocalDbTestContext("TestDatabaseName") // 1
        .DeployDacpac()                        // 2
        .RunTest((connection, transaction) =>  // 3
    {
        // 4
        var order = @"
        | Id | Customers_Id | DateCreated | Product | Quantity | Price | Notes       |
        | -- | ------------ | ----------- | ------- | -------- | ----- | ----------- |
        | 23 | 1            | 2021/07/21  | Apples  | 21       | 5.29  | emptyString |";

        Given.UsingThe(_context)
            .TheFollowingSqlStatementIsExecuted(
                "ALTER TABLE Orders DROP CONSTRAINT FK_Orders_Customers;") // 5
            .And.TheFollowingDataExistsInTheTable("Orders", order); // 6

        When.UsingThe(_context)
            .TheStoredProcedureIsExecutedWithReader("spFetchOrderById", 
                ("OrderId", 23)); // 7

        Then.UsingThe(_context)
            .TheReaderQueryResultsShouldContain(@"| Id | Product |
                                                  | -- | ------- |
                                                  | 23 | Apples  |"); // 8

    });
}

Hopefully, the test is fairly self-explanatory :) but this is what’s going on:

1. the LocalDbTestContext constructor creates a temporary instance of localDb and creates a connection to it.
2. the DeployDacpac method deploys DacPac containing the database schema we want to test.
3. _context.RunTest(...) this is where we define an Action delegate which is the actual test, making use of the supplied connection and transaction.
4. var order = @" here some setup data is defined using the markdown table syntax, this will be parsed into a TabularData object, see the TabularData section below, but you can populate data however you like.
5. Given...TheFollowingSqlStatementIsExecuted() here an arbitrary SQL command is executed, in this case removing a foreign key constraint so we only have to set up the data we specifically need for the test.
6. TheFollowingDataExistsInTheTable() this method takes the order TabularData we just defined and inserts it into the temporary database instance (inside the supplied transaction).
7. When...TheStoredProcedureIsExecutedWithReader() This method executes the named stored procedure that we are trying to test.
8. Then...TheReaderQueryResultsShouldContain() This method asserts that the result returned from the line above contains some data defined in a second tabular data string.
9. After the test is complete its transaction is rolled back leaving the database context unpolluted for the next test.

Leveraging existing work

The LocalDbTestContext class’s constructor is responsible for setting everything up ready for a set of tests to be executed. For managing the LocalDb instances, I am using the excellent MartinCostello.SqlLocalDb package which makes this task relatively trivial and can be found here on GitHub and here on Nuget

For the DacPac deployment, I took inspiration from this StackOverflow thread. The framework contains a class named DacPacInfo which is passed a string either containing a path to a dacpac file or a dacpac file name. If passed a path it will just use that path, otherwise it will traverse up the solution directory structure to a configurable number of levels and then use the file name to search for matching dacpac files and use the first onenit finds, this second method obviously takes longer.

DbTestContext RunTest method

The RunTest() method unsuprisingly runs a test defined in the Action<IDbConnection, IDbTransaction> passed into the method.

A new connection is opened and a new SqlTransaction created which are passed to the Action. The Action is called within a try finally block, which is then used to tidy up any open DataReader objects on the connection and roll back the SqlTransaction after the test Action has been invoked, this ensures that each test starts with a clean slate unaffected by other tests.

There is also an overload of the RunTest() method which not begin and roll back a transaction, this is useful for testing repository classes which usually like to creat a new connection, use it and dispose it soon afterwards. Please not you will have to manage and tidying up of test data to ensure tests do not interfere with each other as theres no transaction to be rolled back for you.

Some nice extra features

TabularData

The TabularData class can be used for human-readable data definition.

We are used to defining tabular data in Markdown tables and also using Specflow’s example tables, data expressed in this format is far easier for a human to ‘parse’ than SQL statements. So, I created a class called TabularData which has methods for converting to and from markdown table strings and also converting to SQL statements, and from SqlDataReader, it also has methods for evaluating whether two TabularData are equal and whether one contains another. The code can be found here. Here are some examples of its use:

// Using markdown/specflow table style strings
var testString = @" | id | state     | created    | ref          |
                    | -- | --------- | ---------- | ------------ |
                    | 1  | created   | 2021/11/02 | 23hgf4hj3gf4 |
                    | 2  | pending   | 2021/11/01 | 623kj4hv6hv4 |
                    | 3  | completed | 2021/10/31 | e0v9736eu476 |";

// Using static builder methods (handy if the table has lots of columns)
var tabularData = TabularData
        .CreateWithColumns("column1", "column2")
        .AddRowWithValues("valueA", "valueB")
        .AddRowWithValues("valueC", "valueD");

String value interpretation

The following table shows how string values are interpreted in TabularData:

One thing to note, the TabularData class doesn’t know the schema of the database table it describes. So if you will be using one to populate a table, the column names and rows data types that you populate it with need to match the table schema otherwise a SqlException will be thrown when attempting to insert the data.

Given, When and Then helper classes

Recently I have started separating out the arrange, act and assert parts of a test into Given, When and Then classes, this gives a nice fluent interface and makes the tests nice and short which improves readability. The framework is un-opinionated about how you interact with the database, but these helper classes just use the System.Data namespace.

The Given class is responsible for the ‘arrange’ part of the test, it contains methods for inserting test data into a table using markdown table strings or an instance of a TabularData. It also contains methods for executing arbitrary SQL statements (e.g. to remove a foreign key constraint to reduce the amount of data seeding required) and

The When class is responsible for the ‘act’ part of the test and contains methods for executing Stored Procedures and various types of query, the results are stored on the shared instance of the LocalDbTestContext so that the Then class can neatly access them for assertions, but there are also overloads which return the result an Out argument.

The Then class is responsible for the ‘assert’ part of the test and contains methods for asserting that query results are equal to or contain data specified using markdown table strings or an instance of a TabularData passed in as an argument. It also has methods for executing either scaler or reader queries in case you need to assert against data in the database changed by a stored procedure under test.

All three share the instance of the IDbTestContext, which allows them to access its State dictionary, its CurrentDataReader and its LastNonReaderQueryResult objects which contain the results of DataReader and non-DataReader queries against the database.

The shared IDbTestContext in these classes is public so you can extend them using extension methods, there is an example of this below.

Here is some of the code for the Given class showing the use of System.Data types to utilise the connection:

public class Given
{
    public ILocalDbTestContext Context;

    public Given(ILocalDbTestContext context)
    {
        Context = context;
    }

    public static Given UsingThe(LocalDbTestContext context) => new(context, logAction);

    public Given And => this; // pointless syntactic sugar to make the tests read nicely
    
    public Given TheDacpacIsDeployed(string dacpacProjectName = "")
    {
        Context.DeployDacpac(dacpacProjectName);

        return this;
    }

    public Given TheFollowingDataExistsInTheTable(string tableName, string markdownTableString)
    {
        var tabularData = TabularData.FromMarkdownTableString(markdownTableString);
        return TheFollowingDataExistsInTheTable(tableName, tabularData);
    }

    public Given TheFollowingDataExistsInTheTable(string tableName, TabularData tabularData)
    {
        try
        {
            var cmd = Context.SqlConnection.CreateCommand();
            cmd.CommandText = tabularData.ToSqlString(tableName);
            cmd.CommandType = CommandType.Text;
            cmd.Transaction = Context.SqlTransaction;

            Context.LastQueryResult = cmd.ExecuteNonQuery();

            LogMessage("TheFollowingDataExistsInTheTable executed successfully");

            return this;
        }
        catch (Exception ex)
        {
            LogMessage($"Exception thrown while executing TheFollowingDataExistsInTheTable, {ex}");
            throw;
        }
    }
    // some methods and the triple slash documentation is removed for brevity    
}

To extend Given to add more methods use extension methods:

public static class GivenExtensions
{
    public static Given SomeOtherSetupOperationIsPerformed(this Given given)
    {
        // do something here
        // e.g. access the shared Context 
        // across the Given, When and Then
        given.Context.State.Add("newStateObject", 87654) 

        // returning this enables the fluent method chaining.
        return given; 
    }
}

Performance and efficiency

Starting a temporary localDb and deploying a DacPac are both quite expensive tasks, so it is best to do these jobs once for a set of tests, various test frameworks achieve this in different ways, in xUnit it is the IClassFixture<T>. A test class that implements this interface will have an instance of T injected into its constructor and the T will be disposed after any tests have been run.

Here is the class that I am injecting using IClassFixture which creates the localDb instance and deploys the DacPac.

public class LocalDbContextFixture : IDisposable
{
    public LocalDbTestContext Context;

    public LocalDbContextFixture(IMessageSink sink)
    {
        Context = new LocalDbTestContext("SampleDb", log => sink.OnMessage(new DiagnosticMessage(log)));
        Context.DeployDacpac(); // If the DacPac name != database name, pass the DacPac name in here, or even better an absolute path to the file.
    }       

    public void Dispose()
    {
        Context.TearDown();
    }
}

In my simple test scenario, the localDb takes around 10 seconds to spin up and the DacPac takes another 10 seconds, but this is roughly comparable to the setup time that an equivalent tSQLt test would take.

Modes

The framework has the following modes:

The mode is set in the constructor of the DbTestContext but it can also be overridden using environment variables:

This means your code can use TemporaryLocalDbInstance by default for local dev but your CI build can switch to a different mode using environment variables.

SQLCover

If you would also like to measure code coverage of your SQL code, there is a great tool named SqlCover which you can read about in Ed Courage’s post Automate SQL Testing using Azure DevOps and SQLCover. If you follow the method in Ed’s post, you would deploy the DacPac during the setup of the SQL container and then use the ExistingDatabaseViaConnectionString and supply the DbTestContext with the connection string to the SQL server in the container. At ClearBank we have a yaml template which takes care of this part for us.

In closing

All of the source code can be found on GitHub here Contributions are very welcome.

So that’s it, a lightweight framework that sets up your db instance and then hopefully gets out of the way, letting you test your db objects and repository classes however you like but also providing a few helpful classes for common tasks. Hope you find it useful and thanks for reading!

Packages are available on Nuget, the main CSharpSqlTests package here, additional Xunit extensions package here and additional NUnit extensions package here

Background

In April 2020 Microsoft brought out a new client library for Azure Service Bus called Azure.Messaging.ServiceBus, this new library is based on the open Advanced Message Queuing Protocol (AMQP) standard. Microsoft outlines:

AMQP enables you to build cross-platform, hybrid applications using a vendor-neutral and implementation-neutral, open standard protocol. You can construct applications using components that are built using different languages and frameworks, and that run on different operating systems. All these components can connect to Service Bus and seamlessly exchange structured business messages efficiently and at full fidelity.

The class library documentation hints at this library being testable! This is great because one of the frustrations of using previous Service Bus client libraries was that absolutely everything was sealed or protected, meaning the only way to test client code was to wrap up and abstract away the service bus infrastructure. The situation might have been helped by a local emulator, but despite being a highly requested and up-voted feature request, an emulator was never officially implemented. So any improvements to the testability of Service Bus client code would be very welcome indeed!

The test support here is basically a few classes which are public and not sealed, a sprinkling of overridable methods and a factory class for creating models.

There are no official samples yet and very few (if any) blog posts covering how to test, but after a bit of trial and error we managed to get some tests working that we are quite pleased with, hence sharing here so that others might also benefit. This article covers testing the very basic handling of messages, although it should also be a useful starting point for testing more advanced features.

Component tests

At ClearBank we are favouring ‘component tests’ over reams of unit tests, I’m not sure who coined the name ‘component test’, but I see them as a kind of internal integration test. So we run some kind of test host, using real concrete objects, with the exception of anything which touches the network, these objects are mocked. We test as much of the service as possible i.e. including the Startup class and the Program class if possible. The tests include happy and sad paths, enough to give sufficiently high line and branch coverage of the tested services. These kind of tests take longer to setup than pure unit tests making Test Driven Development (TDD) a bit harder to start with, but they run locally, they run much faster and are less brittle than integration/end-to-end tests, giving the same confidence that the business logic is proven in a smaller number of tests than traditional unit tests. Of course, you still need a couple of integration tests but these need only prove that the integration is working, rather than all of the paths through the logic.

Testing message handlers

The main requirement when testing a service bus message handler is to be able to create a ServiceBusReceivedMessage. This class doesn’t have a public constructor, but there is ServiceBusModelFactory class with a ServiceBusReceivedMessage() method especially for the purpose! This needs to be at the heart of our test approach.

When we create the artificial ServiceBusReceivedMessage it needs to be packaged up in a ProcessMessageEventArgs however, if we want to spy on what happens to the message, we can inherit from this class and override CompleteMessageAsync, DeadLetterMessageAsync and AbandonMessageAsync methods and set flags accordingly. We called this class TestableProcessMessageEventArgs.

Here is our TestableProcessMessageEventArgs class

public class TestableProcessMessageEventArgs : ProcessMessageEventArgs
{
    public bool WasCompleted{ get; private set; }
    public bool WasDeadLettered{ get; private set; }
    public DateTime Created{ get; }

    public TestableProcessMessageEventArgs(ServiceBusReceivedMessage message) : base(message, null, CancellationToken.None)
    {
        Created = DateTime.Now;
    }

    public override Task CompleteMessageAsync(ServiceBusReceivedMessage message,
        CancellationToken cancellationToken = new CancellationToken())
    {
        WasCompleted = true;
        return Task.CompletedTask;
    }

    public override Task DeadLetterMessageAsync(ServiceBusReceivedMessage message, string deadLetterReason,
        string deadLetterErrorDescription = null, CancellationToken cancellationToken = new())
    {
        WasDeadLettered = true;
        return Task.CompletedTask;
    }

    public override Task AbandonMessageAsync(ServiceBusReceivedMessage message, IDictionary<string, object> propertiesToModify = null,
        CancellationToken cancellationToken = new CancellationToken())
    {
        return Task.CompletedTask;
    }
}

The next thing we need to mock is the ServiceBusProcessor class, this is returned by the ServiceBusClient and communicates with the service bus, passing messages to the handler for a specific queue or topic. We could try to use Moq but its easier to inherit from ServiceBusProcessor and override its StartProcessingAsync() method, this stops it from realising that it isn’t connected to a real service bus. We called this class TestableServiceBusProcessor.

We can then call base.OnProcessMessageAsync(args) on the TestableServiceBusProcessor passing in a an artificial message created using the ServiceBusModelFactory.ServiceBusReceivedMessage() mentioned above. This will present the artificial message to the handler, as if it were connected to a real service bus receiving a real message.

We can add a couple of helpful features to the TestableServiceBusProcessor such as a collection of TestableProcessMessageEventArgs called MessageDeliveryAttempts to assert against and a generic method for sending messages, we can also simulate retries in order to test sad paths and/or exponential back off policies etc.

Here is our TestableServiceBusProcessor class

public class TestableServiceBusProcessor : ServiceBusProcessor
{
    public List<TestableProcessMessageEventArgs> MessageDeliveryAttempts = new();

    public async Task SendMessageWithRetries<T>(T payload, int maxDeliveryCount = 5)
    {
        for (var attempt = 1; attempt <= maxDeliveryCount; attempt++)
        {
            // Don't send retry if the message was sent already and completed
            if (MessageDeliveryAttempts.Any() && MessageDeliveryAttempts.Last().WasCompleted)
                return;

            await SendMessage(payload, attempt);

            // Simulate the message being deadlettered if max delivery count is hit
            if (attempt == maxDeliveryCount)
                MessageDeliveryAttempts.Last().WasDeadLettered = true;

        }
    }

    public async Task SendMessage<T>(T payload, int attempt = 1)
    {
        var args = CreateMessageArgs(payload, attempt);
        MessageDeliveryAttempts.Add((TestableProcessMessageEventArgs)args);
        await base.OnProcessMessageAsync(args);
    }

    public ProcessMessageEventArgs CreateMessageArgs<T>(T payload, int deliveryCount = 1)
    {
        var payloadJson = JsonSerializer.Serialize(payload);

        var message = ServiceBusModelFactory.ServiceBusReceivedMessage(
            body: BinaryData.FromString(payloadJson),
            deliveryCount: deliveryCount);

        var args = new TestableProcessMessageEventArgs(message);

        return args;
    }

    public override async Task StartProcessingAsync(CancellationToken cancellationToken = default)
    {
    }
}

How to use them

So, to use these classes in a component test, we can use the excellent TestHost from the Microsoft.AspNetCore.TestHost package, this enables us to run a service in memory, calling the ConfigureServices method as normal, but then overriding some of the configurations i.e. the ones that touch the network that we need to mock. Consider the following code:

public class TestHost : IDisposable
{
    private readonly IHost _server;
    public Mock<ServiceBusSender> MockTestQueue2Sender { get; } = new();
    public TestableServiceBusProcessor TestableQueue1MessageProcessor { get; } = new();
    public Mock<IWidget> MockWidgetService { get; } = new();
    public IServiceProvider Services => _server.Services;
    public int NumberOfSimulatedServiceBusMessageRetries = 5;
    public int InitialRetryDelayForServiceBusMessageRetriesInMs = 100;

    public TestHost()
    {
        var builder = new HostBuilder()
            .ConfigureWebHost(webHost =>
            {
                webHost.UseTestServer()
                    .ConfigureServices((context, services) => Program.ConfigureHost(services, context.Configuration))
                    .ConfigureTestServices((services) =>
                    {
                        var client = new Mock<ServiceBusClient>();

                        client.Setup(t => t.CreateSender(It.Is < string > (s => s == "testQueue2")))
                            .Returns(MockTestQueue2Sender.Object);

                        client.Setup(t => t.CreateProcessor(It.Is<string>(s => s == $"testQueue1")))
                            .Returns(TestableQueue1MessageProcessor);

                        services.AddSingleton(client.Object);

                        // register other test services here...
                        services.AddSingleton(MockWidgetService.Object);

                    }).Configure(app => { });
            });

        _server = builder.Build();
        _server.StartAsync();
    }

    public void Dispose()
    {
        _server?.StopAsync().GetAwaiter().GetResult();
        _server?.Dispose();
    }
}

So we’re using Moq to create a Mock ServiceBusClient, this will be registered into the DI container and when it’s CreateProcessor() method is called with the correct queue name, an instance of our TestableServiceBusProcessor will be returned. We are also setting up a Mock ServiceBusSender which we can use later to Assert whether messages were sent to a different queue. Finally we have a Mock IWidget service, which our message handler calls and we can use to throw exceptions or simulate processing delays etc.

The actual tests

The following code shows the actual tests, we decided to split the setup, execution and assertion into a set of helpers named Given, When and Then, this allows for a nice fluent interface for the tests and makes them nice and readable.

[Fact]
public void a_message_sent_to_the_queue_is_handled_and_completed()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload);

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload);

    Then.OnThe(host)
        .TheWidgetServiceWasCalled(times: 1)
        .And().AMessageWasSentToTestQueue2()
        .And().TheMessageWasCompleted();
}

[Fact]
public void a_message_sent_to_the_queue_is_handled_and_when_permanentException_is_thrown_is_dead_lettered()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload)
        .And().TheWidgetWillThrowAPermanentException();

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload);

    Then.OnThe(host)
        .AMessageWasSentToTestQueue2(times: 0)
        .And().TheMessageWasDeadLettered();
}

[Fact]
public void a_message_sent_to_the_queue_is_handled_and_when_transientException_is_thrown_is_retried_and_is_eventually_completed()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload)
        .And().TheWidgetWillThrowANumberOfTransientExceptions(times:3);

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload, simulateRetries: true);

    Then.OnThe(host)
        .TheMessageWasRetried(times:4)
        .And().TheWidgetServiceWasCalled(times:4)
        .And().TheRetriedMessagesHadIncreasingDelays()
        .And().AMessageWasSentToTestQueue2(times:1)
        .And().TheMessageWasCompleted();
}

Given, When and Then test helpers

Here is the code for the When class which simulates the message appearing on the queue

public class When
{
    private readonly TestHost _host;

    public When(TestHost host)
    {
        _host = host;
    }

    public static When OnThe(TestHost host) => new(host);
    public When And() => this;

    public When AMessageIsSentToTestQueue1(string testPayload, bool simulateRetries = false)
    {
        var payload = new TestQueueMessage(testPayload);

        if (simulateRetries)
            _host.TestableQueue1MessageProcessor.SendMessageWithRetries(payload, _host.NumberOfSimulatedServiceBusMessageRetries).GetAwaiter().GetResult();
        else
            _host.TestableQueue1MessageProcessor.SendMessage(payload).GetAwaiter().GetResult();

        return this;
    }
}

And here is the code for the Then class which demonstrates how to use the TestableServiceBusProcessor and its collection of MessageDeliveryAttempts, each of which contains a TestableProcessMessageEventArgs that be can used to make assertions. We are using the FluentAssertions package.

public class Then
{
    private readonly TestHost _host;

    public Then(TestHost host)
    {
        _host = host;
    }

    public static Then OnThe(TestHost host) => new(host);
    public Then And() => this;

    public Then TheMessageWasCompleted()
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Last().WasCompleted.Should().BeTrue();
        return this;
    }

    public Then TheMessageWasDeadLettered()
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Last().WasDeadLettered.Should().BeTrue();
        return this;
    }

    public Then TheMessageWasRetried(int times)
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Count.Should().Be(times);
        return this;
    }

    public Then TheWidgetServiceWasCalled(int times) 
    {
        _host.MockWidgetService.Verify(x => x.DoSomething(It.IsAny<string>()), Times.Exactly(times));
        return this;
    }

    public Then AMessageWasSentToTestQueue2(int times = 1)
    {
        _host.MockTestQueue2Sender.Verify(x => x.SendMessageAsync(It.IsAny<ServiceBusMessage>(), It.IsAny<CancellationToken>()), Times.Exactly(times));
        return this;
    }

    public Then TheRetriedMessagesHadIncreasingDelays()
    {
        var delays = new List<TimeSpan>();
        for (var i = 0; i < _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Count - 1; i++)
        {
            delays.Add(_host.TestableQueue1MessageProcessor.MessageDeliveryAttempts[i + 1].Created -
                        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts[i].Created);
        }

        for (var i = 0; i < delays.Count - 1; i++)
        {
            Assert.True(delays[i + 1] > delays[i], $"the interval of retry{i+1}({delays[i+1].TotalMilliseconds}ms) was not larger than the previous retry({delays[i].TotalMilliseconds}ms)");
        }

        return this;
    }
}

And that is how we are testing message handling code which uses the Azure.Messaging.Servicebus client library. Thankyou for reading and hope you find it useful.

All of the code from this post can be found here

The title image is from here

This blog was first published on the ClearBank tech blog

Background

Over the last two or three years, my team has been building a distributed system, a set of dotNetCore microservices using EventStore, Consul and ElasticSearch. There have been times where a certain task needed to completed exactly once even though we had 5 nodes in the cluster, so which node should perform the task? We used advice from a paper in the documentation of Consul, which is a fantastic tool made by Hashicorp. Consul has a concept of sessions and locks, where the nodes in a system can all attempt to place their own session token into the lock, one will succeed and perform the task and the others will wait for their turn.

Whats wrong with having a leader?

Even with the help of Consul, which is designed to help with the problems involved with building distributed systems, leader election code is necessarily complex. Only one node should be leader at any one time. All of the other nodes should know who the leader is and that they themselves are not the leader. If the current leader dies, another node should take over without losing any tasks that should have been performed. We found that, for a time sensitive task, even leaders in waiting needed to listen for and attempt to perform all of the leader’s tasks, just in case they became the leader.

Super basic explanation of Blockchain

Anyway, back to Blockchain! Blockchain is a brilliant concept of a chain of blocks, where each block contains some data, the previous block’s hash and the block’s own hash. There is a brilliant graphical explanation here, but I will attempt my own explanation anyway…

First we have to explain hashes. A hash is a unique value that can be mathematically calculated from a given piece of data using an algorithm. The hash of a given piece of data will always be the same, meaning if you send the data to another place along with the original hash, the arrived data can be hashed again and if the two hashes match, you can be sure that arrived data is identical to the original data that was sent. It is not possible to derive the data from the hash, the hash is like a digital fingerprint of the data.

The first block in the chain is special in that its previous hash is null, its called the genesis block.

Each block’s hash contains the previous block’s hash, or in other words the fingerprint of a block is made up of the data in that block AND the previous block’s fingerprint. Its possible to loop through all of the blocks in the chain checking the hashes and proving that the chain is valid. If the chain is instantiated in two locations and both have the same blocks added, the chains (including all of the hashes) will be identical. If one chain has any differences, however small, the hashes will be different from the altered block onwards, the difference between the two chains will be obvious.

So, as long as there are a number of chains (and preferably an odd number) and less than 50% of them have been tampered with, the system can detect the tampering and reject the altered blocks.

Whats the difference between public and private network Blockchain?

In a crypto currency, there is a large public network of nodes, in fact anyone can run a node and have a copy of the chain. The source of truth is therefore completely decentralised and as long as no one party owns more than 50% of the nodes, the system can be trusted.

Public network Blockchain also has a concept called ‘proof of work’ which makes it non trivial to be able to create the next block, one example is that the hash must start with a certain number of leading zeros, this is where the concept of mining comes from. A number called a nonce is added to the block and if the hash doesn’t meet the constraint (e.g. starting with 11 zeros) the nonce would be incremented and the hash recalculated, this happens repeatedly until the hash meets the constraint. This would probably mean that a special piece of code running on a high-end graphics card on a ‘mining rig’ would have been churning out millions of hashes for about 10 minutes before the next block was found. This is a very simplified explanation, in BitCoin for instance there is also a confirmation process explained here

Its this trust that makes public network blockchain potentially useful for things like currency, banking/financial stuff, supply chain provenance, asset tracking, contracts, medical records etc and loads more.

In a private network, all of the nodes are trusted so there is no need for proof-of-work and the next plain old hash is perfectly fine, which is very fast compared to finding the next hash which starts with 11 zeros.

I feel a side project coming on…

So my idea (and I know I’m probably not the first to have had it) is that a private network Blockchain should be able to enable a distributed system to agree on the state of the truth without needing to elect a leader.

Each node in the distributed system would have its own copy of the Blockchain. Any new data added should be broadcasted to the other nodes, who will attempt to verify their Blockchain with the new block on the end. If they cant verify the chain, they will reject the new block, sending a negative response and the original node will have to try again.

So in a cluster with nodeA, nodeB and nodeC, if new data is added to nodeA and nodeB at the same time, one of them (lets say nodeB) will succeed in sending the new block to nodeC first, the other (nodeA) will (having received the new data from nodeB itself) recreate the new block and try again.

This gets more interesting as the number of nodes increases and my guess is that as long as I know the total number of nodes and therefore the size of a quorum (i.e. the number of nodes that represents more than half of the total) a given node should continue pushing a new block as long as it doesn’t receive more than the quorum amount of negative responses and it should definitely continue pushing a block if it receives more than the quorum amount of positive responses! The larger the cluster the more chance of competition, this is the bit that I am most looking forward to experimenting with.

I needed a concrete use case to test my theory and happened to have 50 odd Gb of family photos which need sorting so my plan was to build a WebApi which will allow me to store and retrieve files (mostly photos) and to define and search on metadata associated with the files I have stored. The files will be stored in an Azure blob store and the metadata will be stored in a Blockchain. The WebApi should run on multiple nodes as a distributed system, where all nodes are capable of reading and writing data. Whenever data is written to a node it should broadcast the new data to the other nodes and therefore all nodes should have an identical Blockchain.

How will I know when the question has been answered?

I guess I will have my system up and running, receiving uploaded photos as fast as possible for a sustained period of time, while being able to simultaneously make read successful requests and metadata searches. At any point in time, I should be able to check the latest block’s hash matches across the cluster.

Progress so far

So far, I have:

• developed the basic dotNetCore WebApi service with Swagger, starting using LiteDb behind an abstraction
• developed my Blockchain implementation with unit tests
• developed a P2P module using the excellent GRPC
• developed a Blockchain database and switched it in place of LiteDB.
• written some NUnit integration tests that add 3000 random strings of data to a random node in a cluster of three, as fast as possible and check that the Blockchains are identical at the end
• persisted the chain to an Azure blob after each write
• added some ridiculously fast indexing using the humble Dictionary class.
• added logging to Elasticsearch
• have centralised configuration in Hashicorp Consul

Next steps

Next, I plan to:

• put the source onto GitHub
• write tests which prove that the cluster does not get into a split brain scenario after two coincident writes
• improve the Blockchain persistence
• have new nodes initialise their Blockchains from the blob store
• periodically check for and prune dead nodes from the list.
• develop an Angular frontend
• add some kind of decent load testing

Future ambitions

• I need an uploader application.
• I want to add indexing, including geospatial metadata, in ElasticSearch
• I want to deploy the thing to Azure Kubernetes Service.
• Long term I’m thinking of using Cognitive services Face Api and maybe some machine learning to fill in missing Tags etc.

Thanks for reading!

Creativity

I think the main reason I love software engineering is the sheer creativity, we start with a defined a bunch of building blocks (a language and base class library etc) then we go about the process of solving problems or realising ideas using a limitless amount of these blocks, sometimes creating higher level reusable blocks, sometimes pulling in someone elses. With cloud scale, the only limits really are your imagination!

From my personal experience, I see a strong link between my obsession with career in software with a love of Lego as a child. I asked for Lego at every opportunity and enjoyed building the official model, but only ever once, after that I would add the pieces to my very large box and build whatever I wanted. (At college, I built a prototype of a racing car in Lego that had working accelerator, brakes, a clutch and a 3 speed gearbox and I had so much Lego that it was all colour coordinated and everything!)

What I’m not sure about is whether the Lego gave me that creativity or just gave expression to it. My son likes Lego but not in the same way I loved it. I see a similar thing with children who love Minecraft.

Anyway, back to music. Music is obviously a hugely creative thing, although slavishly learning piano pieces for an exam is certainly not fun. I learnt trumpet classically to the point where I could read music by sight fairly well, even transposing into different keys, but I was so frustrated that I couldn’t improvise to save my life! I always longed to be able to improvise! I certainly enjoyed playing music in a group where the sheet music was the key to transforming a set of individual solo efforts into something that was more than the sum of the parts. That was definitely still a creative thing, but different to creating something from scratch. Similar to building the official Lego models.

Personal software development practice and personally playing an instrument

At first, when thinking about what to write, I was looking for parallels between personal software engineering practice and personally playing a musical instrument, there are definitely some: practice, research and learning by reading etc, even going to a conference to hear a particular speaker is similar to going to your favourite band’s gig or a clinic with your favourite bass player etc. Also the concept of a music teacher is quite like that of a mentor or senior in a team who has an interest in inspiring the team members to reach their potential by passing on what they have learnt. But I was sure there must be something more compelling that those!

Comparison of a software development team and a band

I switched to considering the similarities of a software engineering team and a band and I think there is much more to learn.

Playing in a band can be one of the greatest things in life. I am going to focus on what it is that makes a great experience of playing a song in a band, when everything works and a lovely piece of music is the result. I’m thinking about a band with guitars and drums rather than say a brass band or orchestra, although I guess the same applies.

The Groove

A song will have a ‘groove’ which is a kind of rhythmic repeating pattern which is established by the drummer and bass player. The pattern will usually feature accents using the Kick and Snare which the bass player also picks up in the notes they play. The groove sets the ‘feel’ of the song and will be influenced by the style of the music and/or other songs or favourite artists. For instance, in Reggae music the groove is usually swung and places a heavy emphasis on the off beat, whereas in a rock groove it is normally played straight and there is a kick on beats 1 and 3 and a snare on 2 and 4. In western music, which is normally structured in 8 bar sections, there would probably be different grooves for different sections of a given song, one for the verses, one for choruses and another for the bridge etc. At the end of a section there would normally be some kind of fill which is where the drummer and/or bass player would play a something slightly different usually ending in a hit on a cymbal. The groove also has a tempo, which is the speed at which the song is played.

In software…

I’m not sure what the groove equates to in software? It could the technology stack chosen for the job or the agile methodology which shapes the process. There should definitely be a rhythm in a software engineering team and it should feel like everyone in the team is pulling together in one direction and with one shared aim. In the same way that groove has influences from styles or other songs or artists, the members of the team will have had past experiences of technologies or design patterns and will hopefully bring in those experiences where appropriate.

Listen to each other and work together

Probably the most important thing when playing music in a group is to listen to each other. Each person should hold the aim of creating some lovely music more highly important than say personal recognition or being so loud that they drown out everyone else. When each player listens to the others they can stay in time, stay grounded on the groove, they can leave space for each other and play things that compliment the other parts. This is what turns of bunch of separate noises into some music. Band members get to know each other well and begin to ‘play off’ each other where the idea of one person can inspire another idea in someone else. Suggesting an idea can be quite a daunting thing to do because it makes you a bit vulnerable, you need to know that the other band members will not reject ideas harshly.

In software…

I think there are striking similarities here, in a software engineering team, members should be working together and aware of what the other members are doing. They should help each other when needed and ensure that when separate streams of work are going on in parallel, the work can be integrated easily when it comes back together.

In terms of suggesting and trying out ideas, this is crucial:

suggestions, questions and contributions to discussion are not only wanted, they are absolutely needed from every member of the team! There are no stupid questions!

I suggest you drill this into each and every member of your team!

The key to working together well is communication. Team members should be able to inspire each other, to respectfully challenge each other, to show flexibility and to negotiate and compromise. You need the culture to be friendly, open and honest for this to happen. I’ve found that admitting your own mistakes, failures and weaknesses really helps with this. You want your team members to be comfortable showing vulnerability, to admit when they don’t know or understand something or to be able to say when they are just having a terrible day. Once this does start to happen, you will have a team who feels ‘safe to fail’ having the emotional security they need to have ideas, make suggestions, try things out, learn from failures knowing no-one is going to punish them etc. Most importantly they will realise that they have a voice and an important contribution to add, which will make them enjoy their job and feel fulfilled.

Dynamics

A good song feels like its going somewhere, it starts in one place and builds to another, there can be quiet sections, sections which build and louder sections.

In software…

In software engineering teams, we tend to focus on a consistently high velocity, I wonder if teams should plan quieter times or sprints with a different focus? I have heard of the concept of ‘firebreak’ sprints where the team is allowed a whole sprint where they can decide what to work on. This might be a piece of tech debt or a new concept idea or some kind of test or deployment tool etc. Even the concept of a ‘sprint’ has always seemed a bit wrong to me - as no-one can keep sprinting forever (however much business like the idea!) its a short exhausting burst. I prefer a decent paced long distance event, which might be more like Kanban, but thats probably a different post.

Practices and performances

If a band has a performance or tour coming up, they will practice and practice and practice some more. They will know their parts so well they could play them in their sleep and then when the time comes they can deliver a perfect performance to their fans.

In software…

I think this could have some relevance in the area of deployments or migrations. I’ve worked in places where deployments were practiced, but rarely in a sufficiently live-like environment to promote serious confidence. Maybe if we thought that people would be watching we would do things differently?

To sum up

I think there are a lot of interesting similarities between bands playing music and software engineering teams, some more obvious and some more subtle. There may be aspects of both which would benefit the other in some way. The analogy of course breaks after a while, take pair programming for instance, that would be very interesting in a band! Thanks for reading.

I never intended to stay at one company for 5 1/2 years, but I did get a promotion from senior to lead 3 1/2 years ago and was committed to staying for the duration of a large project which was likely to last another 2 years. The annoying things at work were winding me up quite a lot, to the point of losing sleep. One particular sleepless night, with everything swirling around in my mind, I was wondering how I would try to find my next company.

I had learned through the course of several past jobs and projects quite a few things that I did and did not like, but the tricky bit seemed to be how would I measure a prospective company against those opinions in the time it takes to have an interview? Then I had the idea of creating my ideal ‘Company Spec’, I sat up in bed, fired up my laptop and started typing, exactly one hour later having dumped all my thoughts, I finally dropped off to sleep, feeling much better.

Company spec

What I meant was a set of points describing what I wanted from my next employer in the same way that an employer would describe what they wanted in their next employee in a job spec. I’ll include my actual company spec below.

Firstly, I got positive comments from recruiters, but also positive feedback from the first interesting looking company that I decided to pursue, in fact I discussed several points from the Company Spec with the CTO of that company in the telephone interview that followed and then the face-to-face interview that followed that! I ended up seriously considering and eventually accepting a job with that same company!

I found that my company spec did several things:

• it let the company know that I have opinions and think things though
• it is source of stories to tell - the reasons I have each opinion.
• its a source of questions for me, as you can always take a copy of your CV into an interview and you can refer to it when your mind inevitably goes blank when they ask if you have any questions.
• it let the company weigh themselves up against the points in my list
• it potentially could have helped a thorough recruiter in not wasting my time and theirs on a job which clearly did not compare favourably with my list.

By the end of the interview and certainly by the time I received an offer, we had discussed all of the points in my company spec in detail and to the point that I was pretty sure it was a company I could work for and I think they were fairly sure I was someone they could employ.

Include an interesting side project

The other thing I did was to place a short section about an interesting side project and what progress I’d made with it. In my case I am convinced that private network blockchain can solve the problem of consensus in a distributed system without needing a leader, so I have a project in which I am trying to prove it. I am also using my project to learn about containers and Azure, both of which I haven’t used much in a commercial context. I supposed a decent open source project would be even better, but any interesting project or problem to solve would work. As long as you can talk about it confidently when the questions come, I talked about my intentions for future architecture and some stretch goals. I think this would be especially useful if the projects you have been working on are not utilising modern architectures or design patterns, as it shows awareness of modern thinking and a bit of initiative on your part to go above and beyond to learn them.

Both of these are really just a way of steering the conversation to something you can talk about enthusiastically

My final CV layout was:

• Name, address, basic info etc
• Summary list of languages/technologies with rough estimate of years/months of experience
• Short section on my side project (don’t hide this bit!)
• My Company Spec
• Work experience
• Education
• Short personal statement

The actual company spec section I included

Probably not best to copy mine as you’ll need to talk about them and the experiences good and bad which lead you to the opinion, but this is what worked for me:

In Conclusion

I’m not saying that my company spec and side project sections in my CV got me my new job, but they certainly helped with the content of the conversations in the two interviews I had and certainly left me feeling confident about the ‘fit’ of the company for me. This was definitely the smoothest recruiter->telephone->face-to-face->offer->negotiation->acceptance process I’ve ever had. I have notice to work before I can start my new job and we will have to wait and see how things go but I’m feeling very excited about it!