My Favorite NServiceBus Features - Message Encryption
One of the common hurtles you will need to cross when introducing NServiceBus to an organization is to gain the confidence of the compliance and security teams. These teams have the very important job to make sure the interests of the company are protected from threats internal and external. This is in no means an easy job and while it can feel like they sometimes are putting up roadblocks, it is important to make sure they understand the technology you are introducing and the protections it brings.
In my experience, messaging systems in general make both the compliance and security teams uncomfortable. Up until recently, messaging systems have not been exactly main stream and I believe part of their discomfort is due to lack of familiarity. Another contributing factor is that as Architects and Developers we have not done a great job of explaining how messaging systems work in a way that addresses their concerns.
The compliance team is typically concerned with understanding the provenance of the data, who has access and who has the ability to modify data. They are also concerned with making sure there are sufficient controls to ensure standard operating procedures are followed consistently. This is especially important in the financial and healthcare realms where not having sufficient controls can open the door to abuse and harm.
The security team is typically concerned with access control from the standpoint of minimizing the ability for a nefarious actor to exfiltrate sensitive data or compromise the integrity of systems.
With this in mind, I'm sure you can imagine what they are thinking when you describe a utopia where systems are loosely coupled and communicate with durable messages that anyone can subscribe to. This sounds like everything they are trying to protect the company from. I imagine that architectural utopia translates to lack of controls and sensitive data laying around waiting to be picked up by anyone from the perspective of the compliance and security teams.
The good news is that while implementation of a solution with the appropriate controls is your responsibility, NServiceBus provides the tools to implement those controls. Knowledge of those tools and the ability to explain those controls to the compliance and security teams will make the conversation go much smoother based on my experience. In fact, once you describe the controls you can implement with NServiceBus, you may find that the compliance and security teams become your biggest supporters.
Message Encryption Challenges
The first step at understanding the tools is to understand the challenges of encrypting messages. With an HTTP endpoint, the data is typically encrypted during transit using TLS/SSL and the data is persisted to a database that resided on storage that is encrypted at rest. This is relatively simple for everyone to get their heads around and the focus is usually on the database controls. When messaging is introduced, we can choose a transport (queuing technology) that includes encryption during transport and at rest.
Where the conversation gets rocky is that while a database is easy to wall off for access, messaging transports tend to be more open since they are really a communication channel and not a data repository. The other challenge is that support staff typically need to inspect any message that ends up in the error queue to determine how to best remedy the error.
Encrypting the contents of the message is the best way to mitigate the risk that a message gets transmitted to an unintended destination or viewed by support staff. Encrypting message contents can also provide some protections against someone manipulating the data outside of the normal controls.
The first challenge you will encounter when you discuss encrypting the contents of the message will be how do you rotate encryption keys. The security team will insist, for good reason, that your solution be capable of rotating keys quickly and with a minimum of effort.
The durable nature of messages means that rotating keys is a challenge. If you do a big bang swap of keys, any older messages will suddenly not process and you will have what is essentially corrupt data to deal with. Draining down the messages in your system to switch keys is not an acceptable solution since there could be messages in the error queue or deferred messages that will process at a later date. Draining down the queues will also not meet the security teams requirement that key rotations are quick and with minimum effort.
The Solution
There are two types of message encryption capabilities NServiceBus provides and they are Body Encryption and Property Encryption. In both of these solutions the data in the message is encrypted and decrypted in the message handling pipeline and is essentially transparent to your business logic.
Body Encryption
Body encryption is implemented with a transport message mutator and provides the most flexibility but also requires the most implementation work. Typically, body encryption encrypts the payload and leaves the headers unencrypted.
As you can see in the example below, the components of the message and how you encrypt/decrypt it is left up to you in this model. This provides a high level of flexibility, but does not have much of an onramp to get you up and running.
public class MessageEncryptor :
IMutateIncomingTransportMessages,
IMutateOutgoingTransportMessages
{
public Task MutateIncoming(MutateIncomingTransportMessageContext context)
{
context.Body = context.Body.Reverse().ToArray();
return Task.CompletedTask;
}
public Task MutateOutgoing(MutateOutgoingTransportMessageContext context)
{
context.OutgoingBody = context.OutgoingBody.Reverse().ToArray();
return Task.CompletedTask;
}
}
While encrypting the entire payload of the message may be appealing, there is a significant downside to this approach. If any message fails and gets moved to your error queue, the only way to inspect the body of the message will be to decrypt it with the key. This means your support staff will need access to the encryption key or will need to escalate to pull in additional administrators who do have access to the key.
Property Encryption
With property encryption, NServiceBus provides more facilities to make your implementation easier from the amount of work and the ability to live with it. Property encryption is by far my preferred solution for implementing message encryption for two reasons.
The first reason is that property encryption leverages conventions. This means any property name that matches a pattern you define is automatically encrypted. For example you could define a pattern that will automatically encrypt any property that includes 'AccountNumber' in the name. These patterns can be aligned with your organizations policies on sensitive data or you could choose to mark every property with sensitive data with 'sensitive' or 'encrypted' in the name.
The second reason property encryption is my preferred option is that it only encrypts the properties you define as containing sensitive data. This makes debugging and tracing much easier to do in production since not all of the data in the message is encrypted. Remember, that the transport you choose for production should already include encryption during transit and at rest. The extra level of encryption we introduce in the message is really protection against data manipulation and obscuring sensitive data from support staff.
In the examples below, you will see a message defined with a property that matches the property encryption convention configured below it.
Message with encrypted property
public class MyMessage :
IMessage
{
public string ReferenceId { get; set; }
//Property will be encrypted by convention
public string AccountNumberEncrypted { get; set; }
}
Endpoint configured with property encryption convention
var encryptionService = new RijndaelEncryptionService(
encryptionKeyIdentifier: "2015-10",
key: Convert.FromBase64String("gdDbqRpqdRbTs3mhdZh9qCaDaxJXl+e6"));
endpointConfiguration.EnableMessagePropertyEncryption(
encryptionService: encryptionService,
encryptedPropertyConvention: propertyInfo =>
{
return propertyInfo.Name.EndsWith("Encrypted");
}
);
Property is encrypted in message
{
"ReferenceId":"4c7da743-9183-44b4-901d-eaa20721c6ad",
"AccountNumberEncrypted":"S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABBdkffqB8lbAWIFOni24SHnvPArVB2jUoxsNnDYdB3UQ3R@"
}
While property encryption is completely customizable to meet your needs, it does work out of the box with a minimum of configuration. This is good from the standpoint of getting you up and running and then allowing you to customize only what you need to customize.
Often organizations have a standard encryption library that is mandated or set of algorithms that are specified by the security team. The ability to easily implement your own NServiceBus IEncryptionService and configure it as part of your endpoint makes this straight forward to implement.
Key Rotation
One of my main assertions above was that key rotation is a challenge when considering the encryption of the contents of a message. Since messages are durable and not temporally coupled, you will have messages that were encrypted using the expired key.
NServiceBus Property Encryption has a pattern for solving this issue that leverages a key ring and a message header to allow for the decryption of older messages. The key ring is a collection of keys that you store in a persistence that you implement. The message header contains a key identifier that was used to encrypt the message. The identifier in the message header is used to look up the key in the keyring and decrypt the message accordingly.
The example below from the NServiceBus documentation illustrates the conceptual components of this key rotation pattern. This simplistic example should never be used for production, but it does show an example of the components required.
//Identifier for the key that will be used to encrypt messages going forward
var defaultKey = "2015-10";
//Keyring of keys used to decrypt older messages and encrypt messages going forward
var keys = new Dictionary<string, byte[]>
{
{"2015-10", Convert.FromBase64String("gdDbqRpqdRbTs3mhdZh9qCaDaxJXl+e6")},
{"2015-09", Convert.FromBase64String("abDbqRpQdRbTs3mhdZh9qCaDaxJXl+e6")},
{"2015-08", Convert.FromBase64String("cdDbqRpQdRbTs3mhdZh9qCaDaxJXl+e6")},
};
var encryptionService = new RijndaelEncryptionService(defaultKey, keys);
endpointConfiguration.EnableMessagePropertyEncryption(encryptionService);
This same pattern can be used in the message body encryption mutator, however the implementation of this pattern will be completely up to you.
In Summary
Message encryption is an important concept to understand and be able to communicate to your security and compliance teams. It addresses some of the key risk vectors they are protecting your organization from and will help them understand that message based systems can be implemented in a way that does not expose your organization to undue risk.
Property encryption is one of my favorite NServiceBus features in that it provides a pattern for implementing message encryption as well as providing a concrete basis for addressing the concerns of compliance and security teams. Additionally, once your endpoint conventions are configured, it is easy to live with since it gets implemented only where necessary and in a consistent way.
Additional Resources
You May Also Like
Convert HTML into JSON using Semantic Kernel and OpenAI
