Using secrets in Gallium Data

Introduction

When writing filter logic in Gallium Data, it's not unusual to need access to some secrets: passwords, keys, that sort of thing.

By now, we all know that embedding secrets in code is just not a good idea, though it still happens thousands of times every day. That's why lots of smart people have come up with various ways of managing secrets. In this article, we'll take a look at a few examples.

Docker secrets

Gallium Data typically runs as a Docker container, so the most obvious way to handle secrets is with Docker secrets. This is only available in Docker Swarm, so if you're not using that, this is not an option for you.

You create a Docker secret with a simple command, e.g.:

echo E46159B21D98EDEA1BB9F924B7A92A56 | docker secret create secret_key_1 -

When starting Gallium Data as a Docker service, you then specify that it should have access to that secret:

docker service create --name gallium --secret secret_key_1 [...] galliumdata/gallium-data-engine:x.x.x-yyy

This exposes the secret as a file in the container, in this case /run/secrets/secret_key_1

You can then easily read that secret in your filter code, for instance:

let JavaFiles = Java.type("java.nio.file.Files");

let JavaPaths = Java.type("java.nio.file.Paths");

let keyStr = JavaFiles.readAllLines(JavaPaths.get("/run/secrets/secret_key_1"))[0];

That's all there is to it. Obviously, you'd want to do some error handling, for instance if the secret file is not found, or does not contain what you expected.

Kubernetes Secrets

Beyond just Docker, it's common to run Gallium Data instances in Kubernetes, which offers a Secrets service that's quite similar to Docker's. Kubernetes can expose secrets either as files, like Docker, or as environment variables, which work equally well. There are other, more complex ways of doing this with kubelets too.

There are multiple ways create a secret in Kubernetes, the simplest is with a command:

kubectl create secret generic secret_key_1 --from-literal=secret_key='E46159B21D98EDEA1BB9F924B7A92A56'

Once that's done, the secret can be exposed to your Gallium Data instances by declaring it in your pod definition (in the spec section):

volumes:

- name: secretVol

secret:

secretName: secret_key_1

The container can then read the secret from file /etc/secretVol/secret_key_1

Exposing the secret as an environment variable is not all that different, you also declare it in your pod definition:

env:

- name: SECRET_KEY

valueFrom:

secretKeyRef:

name: secret_key_1

key: secret_key

and you can read it from your filter code with something like:

let JavaSystem = Java.type("java.lang.System");

let keyStr = JavaSystem.getenv("SECRET_KEY");

If you need to change the secret, the easiest thing to do is usually to just restart the pod, but there are ways to get notified of the change -- see the Kubernetes documentation for details.

Azure Secret Vault

If you live in the Windows world, then there's a good chance you're using Azure in one way or another. Microsoft offers an excellent secret management solution called Secret Vault, which gives you (and your enterprise managers) a great deal of flexibility.

Creating the secret in Secret Vault is easy, it can be done using the GUI, or using the Azure command line interface (assuming you've already created a vault):

az keyvault secret set --vault-name "MyVault" --name "secret_key_1" --value "E46159B21D98EDEA1BB9F924B7A92A56"

Accessing the secret from your filter code requires that you add Microsoft's Azure library to your Gallium Data repository:

You'll need two libraries: com.azure/azure-security-keyvault-secrets and com.azure/azure-identity.

In your filter code, you can then authenticate with Azure with a service account and retrieve the secret with something like:

let ClientSecretCredentialBuilder = Java.type("com.azure.identity.ClientSecretCredentialBuilder");

let clientCertificateCredential = new ClientSecretCredentialBuilder()

.clientId("abc12345-6789-abcd-fedc-987f65443100")

.clientSecret("abcdef123456abcdef123456abcdef123456")

.tenantId("987654321-4321-1234-8765-1232123212")

.build();

let SecretClientBuilder = Java.type("com.azure.security.keyvault.secrets.SecretClientBuilder");

let client = new SecretClientBuilder()

.vaultUrl("https://acme1.vault.azure.net/")

.credential(clientCertificateCredential)

.buildClient();

let retrievedSecret = client.getSecret("secret_key_1");

let secret = retrievedSecret.getValue();

Note that you may want to externalize some of these Azure authentication bits using Docker secrets or Kubernetes Secrets -- these are not exclusive!

AWS Secrets Manager

A similar service exists in most cloud platforms. Amazon AWS offers a Secrets Manager, which has a lot in common with Azure's Secret Vault.

You create a secret either by using the GUI, or by creating a JSON file:

{

"secret_key": "E46159B21D98EDEA1BB9F924B7A92A56"

}

and using the AWS command line:

aws secretsmanager create-secret --name production/secret_key_1 --secret-string file://secret.json

Just like with Azure, you can then authenticate with AWS by installing their library in your Gallium Data repository:

When is a good time to retrieve secrets?

Retrieving secrets from Azure, AWS or similar cloud services can often take some time -- a second or two is not unusual. A good strategy is to retrieve the secret(s) in a connection filter, which is guaranteed to execute before anything else happens. That filter can then cache the result in the project context:

if (context.projectContext.secrets) {

return;

}

let secrets = context.utils.createObject();

// Retrieve the secrets here

secrets.key1 = mySecret;

context.projectContext.secrets = secrets;

Notice the use of context.utils.createObject() -- this is because JavaScript objects cannot be cached.

That way, the first connection pays the penalty for retrieving the secret, but all subsequent connections have instant access to it.