My Favorite NServiceBus Features - Message Headers

My Favorite NServiceBus Features - Message Headers
by Brad Jolicoeur
09/04/2020

There are a number of subtle but important aspects of message based systems that are not apparent until after you have started building one and running it in production. One of those aspects is message headers. Message headers contain metadata about each message that will help you correlate, route and trace the history of a message in your system. Without this metadata production operations is at a significant disadvantage and identifying and resolving issues will be difficult. From my observations, this metadata seems to consistently be something that is missed in custom implementations of message based systems.

Once metadata deficient systems are in production it is quickly realized that there are challenges in operating the system. This especially true once the type of messages in a system expands beyond a handful. It becomes difficult to know what activity triggered a message and what it means if a message failed. In my opinion, this is one of the key reasons message based systems have a reputation for being difficult to operate.

Surprisingly, I have worked with more than one team that is using NServiceBus that did not realize that the message headers existed or what value they added. In their defense, NServiceBus does a good job of keeping those details in the background until you need them. This is a good thing from the aspect of throwing you into the pit-of-success, but also from the aspect of keeping that noise out of your business logic. To be honest, I didn't pay much attention to message headers until after I had to operate my first NServiceBus implementation in production.

NServiceBus Features That Use Message Headers

I find that understanding what features NServiceBus leverage message headers helps to better understand NServiceBus message headers and what value they add. Remember that NServiceBus is open source and you can go look up the code that creates or consumes these headers to get a detailed understanding of what is happening.

Spoiler alert! If you want NServiceBus inner-workings to remain a mystery, you should probably stop reading this section.

Here are some of the NServiceBus features rely on message headers:

How Do Message Headers Work

NServiceBus adds headers automatically when messages are sent or published. This happens as a mutator in the NServiceBus pipeline and do not require any special configuration to implement.

When a message is consumed by an NServiceBus endpoint, the headers are automatically consumed and used to influence the behavior of the consumption of the message. For example, if property encryption is used, the encryption handler will have the key name available to do the decryption operations before your message handler is executed.

Below is an example of an NServiceBus message sent using SQS as the transport.

aws --endpoint-url=http://localhost:4576 sqs receive-message --queue-url http://localhost:4576/123456789012/Example-PaymentSaga
{
    "Messages": [
        {
            "MessageId": "56e4bb5c-f645-4f48-b5cb-769ec0cf1b0d",
            "ReceiptHandle": "56e4bb5c-f645-4f48-b5cb-769ec0cf1b0d#e18f91be-5df0-4583-83b4-1679aa225e03",
            "MD5OfBody": "c185917fdbf58a53b7e417fd7993cf38",
            "Body": "{\"Headers\":{\"NServiceBus.MessageId\":\"83ddb66b-8ef7-4571-b5cf-abb80100db40\",\"NServiceBus.MessageIntent\":\"Send\",\"NServiceBus.ConversationId\":\"b8fce273-e5c1-4156-862e-abb80100db4b\",\"NServiceBus.CorrelationId\":\"83ddb66b-8ef7-4571-b5cf-abb80100db40\",\"NServiceBus.OriginatingMachine\":\"LAPTOP-3ACB4PKX\",\"NServiceBus.OriginatingEndpoint\":\"Example.WebApp\",\"$.diagnostics.originating.hostid\":\"cbf2c09b3652d8c563d48395a8b58c5d\",\"NServiceBus.ReplyToAddress\":\"Example-WebApp-InstanceID\",\"NServiceBus.RijndaelKeyIdentifier\":\"bc436485-5092-42b8-92a3-0aa8b93536dc\",\"NServiceBus.ContentType\":\"application/json\",\"NServiceBus.EnclosedMessageTypes\":\"Example.PaymentSaga.Contracts.Commands.ProcessPayment, Example.PaymentSaga.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null\",\"NServiceBus.Version\":\"7.2.0\",\"NServiceBus.TimeSent\":\"2020-05-11 15:35:11:356034 Z\"},\"Body\":\"77u/eyJSZWZlcmVuY2VJZCI6IjRjN2RhNzQzLTkxODMtNDRiNC05MDFkLWVhYTIwNzIxYzZhZCIsIkFtb3VudCI6MTAwLjQ1LCJBY2NvdW50TnVtYmVyRW5jcnlwdGVkIjoiUzJGeWJqcGhkM002YTIxek9tVjFMWGRsYzNRdE1qb3hNVEV4TWpJeU1qTXpNek02YTJWNUwySmpORE0yTkRnMUxUVXdPVEl0TkRKaU9DMDVNbUV6TFRCaFlUaGlPVE0xTXpaa1l3QUFBQUJCZGtmZnFCOGxiQVdJRk9uaTI0U0hudlBBclZCMmpVb3hzTm5EWWRCM1VRM1JAIiwiUm91dGluZ051bWJlciI6IjU1NTU1NSIsIlJlcXVlc3REYXRlIjoiMjAyMC0wNS0xMVQxNTozNToxMS4xMzc2NzUzWiJ9\",\"S3BodyKey\":null}"
        }
    ]
}

In this message you will find the following headers and values

  • ConversationId : b8fce273-e5c1-4156-862e-abb80100db4b
  • CorrelationId : 83ddb66b-8ef7-4571-b5cf-abb80100db40
  • RijndaelKeyIdentifier : bc436485-5092-42b8-92a3-0aa8b93536dc
  • EnclosedMessageTypes : Example.PaymentSaga.Contracts.Commands.ProcessPayment
  • TimeSent : 2020-05-11 15:35:11:356034 Z
  • ReplyToAddress : Example-WebApp-InstanceID
  • OriginatingMachine : LAPTOP-3ACB4PKX

You can find the full list of possible NServiceBus message headers and what they are used for in the NServiceBus documentation.

If you have worked with a message based system it is going to be apparent that this data is all going to be valuable in tracing issues when they occur. If you are new to message based systems, take note, it will become apparent very quickly.

Customization of Message Headers

As with almost every feature in NServiceBus, message headers can be customized if you find the standard headers do not meet your needs. Maybe you interface with a system that has it's own correlation id or there is an end-to-end timestamp your system uses for metrics.

Message headers can be manipulated anywhere you have access to a message context including the following:

This is an example showing how to read a message header value within a message handler.

public class ReadHandler :
    IHandleMessages<MyMessage>
{
    public Task Handle(MyMessage message, IMessageHandlerContext context)
    {
        var headers = context.MessageHeaders;
        var nsbVersion = headers[Headers.NServiceBusVersion];
        var customHeader = headers["MyCustomHeader"];
        return Task.CompletedTask;
    }
}

And an example of writing a message handler when sending a message within a message handler.

public async Task Handle(MyMessage message, IMessageHandlerContext context)
    {
        var sendOptions = new SendOptions();
        sendOptions.SetHeader("MyCustomHeader", "My custom value");
        await context.Send(new SomeOtherMessage(), sendOptions)
        ...

While these examples show how easy it is to access the message headers in NServiceBus, consider that there are some downsides to message headers that I will highlight later in this article. Manipulating message headers is something that should be considered carefully.

Also note that since headers are something that you are going to write and consume between multiple endpoints a shared behavior or mutator are better places to work with the message headers over putting that logic in each message handler. This will ensure every endpoint implements the headers in a consistent way.

There is a lot of great information on how to manipulate message headers in the NServicebus documentation.

Using Message Handlers to Debug

As I have eluded, the message headers are extremely useful when debugging your message based system. The most convenient way to see the message headers is to use ServiceInsights or ServicePulse. Both of these tools will provide a user friendly view of the data in the headers and ServiceInsights will also provide you with the raw message header details.

If you do not have access to ServiceInsights or ServicePulse or you prefer to look at the messages directly, you are not out of luck. Using a tool like QueueExplorer to observe the messages in your queues will allow you to view the raw message headers.

Some information that can be determined using NServiceBus message headers include:

  • Viewing the exception stack trace for a message that had a fatal failure
  • Find out which host the message failed on
  • How many times has a message failed and when did it start failing
  • Determine which endpoint and host that generated a message
  • Understand when a message was sent and what endpoint sent it
  • Identify the message contract(s) contained within the message

Downsides to Message Headers

While there is a world of good that comes from message headers there are also a few downsides that need to be understood and considered.

  • Message size is increased by headers
  • Compatibility between NSB and Non-NSB endpoints

Message Size

Almost every message transport has a message size limitation and even if they do not have a small message size limitation, transports often have degraded performance when message size start getting larger. For this reason, keeping your message sizes as small as is practical is important.

As you can see in the example message above, there is a lot more data in your message other than just the payload body. This additional data has the potential to increase your message size. Use care when adding custom headers to your messages and balance the increased message size with the value that header is providing.

The headers should also be considered when you are calculating the size of your payload data and determining if you need to leverage Databus for your messages. The headers plus your payload make up the total message size.

System Compatibility

If your overall system happens to have a mix of endpoints that use NServiceBus and some that do not, these message headers can make things somewhat inconvenient. I say somewhat, because it is not an impossible situation, you just end up doing a little more work.

NServiceBus has no issues consuming messages that do not have the standard headers. It will happily read the message and process it to the best of it's ability. The most common message serializers, XML and Json, can infer the the message type from the message body. If you are using a serializer that is not capable of infering the type, you will need to have the sender include the NServiceBus.EnclosedMessageTypes header so that NServiceBus can deserialize the message and invoke the correct message handlers.

Be aware that any NServiceBus functionality like property encryption and DataBus will not be available to you for messages that do not include headers. You can get around this by having the non-NSB endpoints format messages with the critical NServiceBus headers.

Since the NServiceBus Message Headers are conceptually just a dictionary and there is no real magic to the values in the headers, it will be relatively easy to create your own headers if you need to.

Message Header Misuse

One thing you should never use message headers for is to transmit data that should be in the payload. For instance, don't put the order amount in all the messages associated with an order in a header. Order amount is data associated with a business transaction and should be part of the payload.

Message headers should be reserved for metadata about your message and system. Adding data that should be in the payload to the headers will cross a conceptual boundary and make it more difficult to operate your system in a couple of different ways.

Putting payload data in the headers will make compatibility between endpoints more difficult since you moved something that should be in a message contract into metadata. In addition, you will end up putting specialized business logic into behaviors and mutators where only cross cutting logic should live.

Summary

Message headers are a key concept that are critical to the success of your message based system. Without message headers you will be blind to the context of the messages your system is processing and debugging issues or monitoring the performance of your system will be close to impossible. NServiceBus throws you into the pit-of-success by automatically including message headers into your messages before you realize you actually need the to operate your system effectively in production. This is why it is on my list of favorite features of NServiceBus.

Additional Resources

You May Also Like