Transparent encryption
in Redis

Introduction

This is an advanced tutorial: if you have never used Gallium Data before, we highly recommend you go through the basic tutorial first.

This is a step-by-step tutorial to implement transparent data encryption in Redis using Gallium Data. The same technique can be used for other databases with minimal changes (see for example the column-level encryption tutorial for Microsoft SQL Server).

Setup

We'll be using two Docker containers:

  • one for Redis

  • one for Gallium Data

As a database client, we'll be using redis-cli from inside the Redis container, but we will connect to Redis through Gallium Data.

Start the Docker containers

Run the following from a command line:

docker network create gallium-demo-net

docker run -d --rm --name gallium-
redis --network gallium-demo-net galliumdata/redis-tutorial:3

docker run -d --rm --name gallium-data --network gallium-demo-net -p 8089:8080 -e repository_location=/galliumdata/repo_
redis galliumdata/gallium-data-engine:1.3.0-1217

Once everything is up and running,

Run the following from a command line:

docker exec -it gallium-redis redis-cli -h gallium-data

You will get a Redis prompt:

gallium-data:6379>

Just to make sure everything is working properly, you can retrieve a sample object with:

lindex "Canada:Main Cities" 2

which should return something like:

"Vancouver, BC"

Everything is now in place, let's get started.

Creating a key

To encrypt and decrypt data, we'll need a secret key. Rather than creating it many times, we can create it once in a connection filter, and reuse it from other filters.

Connect to Gallium Data at http://localhost:8089/

Create a new connection filter of type JavaScript connection filter - call it "Create encryption key"

Set its code to:

if (context.projectContext.crypto) {

return;

}


// Create the private key, and look up a few Java classes we'll use later

let keyStr = "BMl5BYx/FSV8rOQF1IcwzDNRju0bZ7BNKTk584s5Dfk";

context.projectContext.crypto = context.utils.createObject();

context.projectContext.crypto.Base64 = Java.type("java.util.Base64");

let keyBytes = context.projectContext.crypto.Base64.getDecoder().decode(keyStr);

let SecretKeySpec = Java.type("javax.crypto.spec.SecretKeySpec");

context.projectContext.crypto.key = new SecretKeySpec(keyBytes, "AES");

context.projectContext.crypto.Cipher = Java.type("javax.crypto.Cipher");


// Compute the key's hash, but we keep only the first 8 bytes

let MessageDigest = Java.type("java.security.MessageDigest");

let hashBytes = MessageDigest.getInstance("SHA-1").digest(keyBytes);

let Hex = Java.type("org.apache.commons.codec.binary.Hex");

context.projectContext.crypto.keyHash = Hex.encodeHexString(hashBytes).substring(0, 16);

This code creates a private key using Java classes, and computes a hash for the key, which we'll use to indicate which key a value is encrypted with.

Note that the private key is embedded in the code, which is not something you want to do in production. See Using secrets in Gallium Data for pointers on this topic.

Encrypting on insertion

We're going to encrypt only values whose key starts with Secret:. We'll just handle string values, but it would be easy to extend this to other types such as lists, hashes, sets, etc...

⇨ Create a new request filter of type JavaScript request filter - Redis - call it "Encrypt secret data"

Set its Command patterns parameter to:

regex:set

regex:secret:.+

This will ensure this filter gets called only for SET commands, with a key that starts with Secret:

Set the code to:

let cipher = context.projectContext.crypto.Cipher.getInstance("AES");

cipher.init(context.projectContext.crypto.Cipher.ENCRYPT_MODE, context.projectContext.crypto.key);

let valueBytes = context.utils.getUTF8BytesForString(context.packet[2].string);

let encryptedBytes = cipher.doFinal(valueBytes);

let encryptedStr = context.projectContext.crypto.Base64.getEncoder().encodeToString(encryptedBytes);

context.packet[2].string = "crypt:" + context.projectContext.crypto.keyHash + ":" + encryptedStr;

This code will take a Redis command like:

SET Secret:MySecret "This is secret"

and rewrite it to:

SET Secret:MySecret crypt:c9a7ca19ff2f39be:zgwuydpIb68iuWaa3et5ZxZSovfA==

Publish to Gallium Data

Exit redis-cli by running the exit command

Restart redis-cli with:

docker exec -it gallium-redis redis-cli -h gallium-data

We have to restart redis-cli, just this once, because we need to open a new database connection for the connection filter to be invoked.

Run the following in redis-cli:

SET Secret:MySecret1 "This is secret"

SET NotSecret:Alpha "This is not secret"

Now let's make sure the data is indeed stored in Redis.

Run the following in redis-cli:

GET Secret:MySecret1

with a result that looks like:

"crypt:c9a7ca19ff2f39be:zgwuydpIb68iuWaa3et5ZxZSovfA=="

The value has indeed been encrypted on insert, with a prefix of crypt:, followed by the hash of the secret key, so we know which key the value was encrypted with, and finally the encrypted value.

Let's make sure this works only for secret data.

Run the following in redis-cli:

GET NotSecret:Alpha

with the result:

"This is not secret"

So now we know that secret data (with a key starting with Secret:) will be encrypted on insert, but non-secret data will not.

Decrypting on retrieval

Obviously, we don't want clients to see the encrypted value, but rather the real value.

Let's decrypt it automatically when it's retrieved by the clients.

⇨ Create a new response filter of type JavaScript response filter - Redis - name it "Decrypt secret data"

⇨ Set the Command patterns parameter to:

regex:get

regex:Secret:.+

This will ensure that this filter gets called any time someone does a GET for any key that starts with Secret:

⇨ Set the Response patterns parameter to:

regex:crypt:.+

That way, we will decrypt the value only if it's actually encrypted.

⇨ Set the code to:

let noPrefix = context.packet.string.substring(6);

let colonIdx = noPrefix.indexOf(":");

let keyHash = noPrefix.substring(0, colonIdx);

if (keyHash !== context.projectContext.crypto.keyHash) {

throw "Value was encrypted with a different key";

}

let encr = noPrefix.substring(colonIdx + 1);

let encrBytes = context.projectContext.crypto.Base64.getDecoder().decode(encr);

let cipher = context.projectContext.crypto.Cipher.getInstance("AES");

cipher.init(context.projectContext.crypto.Cipher.DECRYPT_MODE,

context.projectContext.crypto.key);

let decrBytes = cipher.doFinal(encrBytes);

context.packet.string = context.utils.stringFromUTF8Bytes(decrBytes);

This will check that the private key used to encrypt the data is the one we are currently using, and decrypt the value.

Publish to Gallium Data

Run the following in redis-cli:

GET Secret:MySecret1

with the result:

"This is secret"

You can convince yourself that the value is still encrypted in the database:

⇨ In Gallium Data, deactivate the Decrypt secret data filter

Publish

Run the following in redis-cli (hint: up-arrow usually works too)

GET Secret:MySecret1

with the result:

"crypt:c9a7ca19ff2f39be:zgwuydpIb68iuWaa3et5ZxZSovfA=="

Conclusion

This is a fairly easy way to add encryption to your Redis database in a way that is completely transparent to the clients -- all they have to do is connect to Gallium Data instead of connecting directly to Redis.

There are many ways in which this could be extended.

For instance, in this tutorial, we do not handle the case where a key is renamed in a way that should encrypt or decrypt its value, but that should be easy: you just need to intercept the RENAME command and encrypt or decrypt the value as needed.

We could handle multiple private keys fairly easily, and if we want to get fancy, we could even re-encrypt values with a new key when they are read.

We could encrypt just certain parts of the value, especially if it's a complex object.

We could make decryption dependent on who the current user is, or their IP address, or the time of day... Your imagination is the limit.

Cleanup

Once you're done with this tutorial, and you want to remove everything that was installed,

⇨ Exit from redis-cli with exit

⇨ Execute the following commands from a command line:

docker stop gallium-redis
docker stop gallium-data
docker network rm gallium-demo-net

This will stop all the Docker containers started during this tutorial.

If you also want to remove the Docker images:

docker rmi galliumdata/redis-tutorial:3
docker rmi galliumdata/gallium-data-engine:1.
3.0-1217

This will remove everything downloaded by this tutorial.