Tutorial 

(SQL Server)

This tutorial will show you how to run Gallium Data on your machine, along with SQL Server. Nothing will be installed permanently, it’s all done with containers, so you can throw everything away when you’re done.

There are several versions of this tutorial: 

The concepts are the same, pick the database you're most familiar with.

This tutorial will take about 10 minutes, depending on your download speed. It can be run on any Intel/AMD machine that runs Docker. Unfortunately, this tutorial does not work on ARM platforms such as the newer Macs (M1/M2) because Microsoft has not released an ARM image of SQL Server yet.

You can also watch this tutorial as a (slightly outdated) video (below, 5 minutes 23 seconds).

Step 1: Make sure Docker is running

For this tutorial, we will use Docker (1). To verify that Docker is indeed available:

⇨ run the following command from a command line:

docker version

You should see some output similar to this:

Client: Docker Engine - Community
 Cloud integration  1.0.33
 Version:           24.0.2
 API version:       1.43
 etc...

The exact version numbers are not important -- the important thing is that Docker needs to be running. 

If you get an error message, you'll need to get Docker up and running before you can continue with this tutorial. Fortunately, there are lots of resources that can help you with that.

We’ll be starting three Docker containers — there are easier ways of doing this using e.g. Docker Compose or Kubernetes, but for this tutorial we want to make sure every component is visible and clearly understood.

We’ll be running all containers in their own Docker network, so let’s create that network. 

⇨ Run the following command:

docker network create gallium-net

The response should be a long string of letters and numbers, which you can safely ignore, something like:

9ef282f6d3cce819a etc...

If you get an error, it may be because you've run another Gallium Data tutorial before, in which case you can ignore the error and carry on.

Docker is now ready, let’s move on to the next step.

Step 2: Start the three containers

If this is your first time running this tutorial, note that it will download about one gigabyte of container images, which can take a while on slower connections.

1 - Start the SQL Server database 

⇨ Run the following from a command line:

docker run -d --rm --name gallium-mssql --network gallium-net galliumdata/gallium-data-demo-mssql:4

This may take a minute as Docker downloads the image and starts it up. 

This image is simply the standard SQL Server image from Microsoft, preloaded with a small sample database. This happens to be the Linux version of SQL Server 2019, but Gallium Data works equally well with the Windows versions of SQL Server.

There is nothing special about this image:  Gallium Data can run with any SQL Server database (2012 and above), including in the cloud (AWS, Azure, etc...)


2 - Start Gallium Data

⇨ Run the following from a command line:

docker run -d --rm --name gallium-data --network gallium-net -p 8089:8080 -e repository_location=/galliumdata/repo_mssql galliumdata/gallium-data-engine:1.8.5-1867

Again, this may take a minute. This is the standard Gallium Data image, with a demo repository, which is set up for this tutorial. In the real world, you will typically use additional options to create your own repository.


3 - Start the database client

Finally we’ll also need a database client. Here we’ll be using DBGate, but any other SQL Server client would work equally well (like Microsoft's SQL tools, or SQL Server Management Studio if you're on Windows). If you want to use one of these other tools, though, you will have to expose port 1433 in the Gallium Data container.

⇨ Run the following from a command line:

docker run -d --rm --name gallium-dbgate --network gallium-net -p 3014:3000 -e CONNECTIONS=db -e LABEL_db='SQL Server' -e SERVER_db=gallium-data -e USER_db=sa -e PASSWORD_db=Password1 -e ENGINE_db=mssql@dbgate-plugin-mssql dbgate/dbgate

This is the standard DBGate image, with options to connect to Gallium Data.

We now have three Docker containers running, speaking to each other as follows:

Step 3: Take a look at the database

⇨ Connect to DBGate at: http://127.0.0.1:3014 

You can browse the GalliumDemoDB database, which is a very small sample database we'll use for the rest of this tutorial.

In particular, notice that some products have a status of discontinued -- this is about to become relevant.

Everything is now in place, now let's see Gallium Data in action.

Step 4: Let's change how the database works

A simple demonstration of the power of Gallium Data can be seen with just a basic filter that will modify certain types of requests. 

⇨ Connect to Gallium Data at: http://127.0.0.1:8089

⇨ Log in

⇨ Open the project named Simple Demo - SQL Server

⇨ Expand the Request filters area

⇨ Open the filter named Filter out discontinued products

Note that the parameters are set so that it executes whenever a simple query is run agains the products table. As it turns out, DBGate executes SQL statements via a stored procedure (sp_executesql), which is not unusual with SQL Server. If it was just using a straight SQL call, we could filter that too (there is a filter called Change SQL to hide some products already defined but disabled -- take a look if you're interested).

⇨ Select the Code tab

The code simply rewrites the query to exclude discontinued products.

⇨ Select the Active checkbox

⇨ Click Publish (top)

⇨ Go back to DBGate and click Refresh (bottom left) to refresh the list of products

Note that all discontinued products have now disappeared. You have changed the behavior of the database without changing either the database or the database client.

Feel free to experiment with the code. Any time you make a change, you'll need to click Publish (top right) to deploy your changes.  For instance, you can hide the value of the status column with:

let param = context.packet.parameters[0];
let regexp = /\[basetbl\].\[status\] AS \[status\]/g;
if (param.value.match(regexp)) {
    param.value = param.value.replace(regexp, "'??' as status");
    log.debug("Edited SQL to: " + param.value);
}

We can also reject queries based on any desired condition, for instance:

if (new Date().getDay() === 0) {  // 0 == Sunday, 1 == Monday, etc...
    context.result.success = false;
    context.result.errorMessage = "You can't query products on Sunday";

    context.result.errorCode = 12345;

    log.debug("Rejecting query because it's Sunday");
}
// Caution: JavaScript may return a UTC date

Any changes in Gallium Data take effect as soon as you hit the Publish button. You can use the log object to print out debugging messages, which you can see in the Logs page of Gallium Data, or by running the following in a command line:

docker logs gallium-data

You can also use the debugger to stop execution and step through your code.

Step 5: Changing data

Now let’s make Gallium Data modify data that's coming back from SQL Server.

⇨ Look at the contents of the customers table in DBGate:

Let's say we don't want our database users to see the full first name of our customers, but just their initial: we can easily make that happen with a response filter.

⇨ Go back to the Gallium Data admin app
⇨  Expand the Response filters area
⇨ Click on the response filter called Show customers initials 

You'll see that it has some parameters that activate this filter for result sets that include data from the gallium_demo.customers table.

⇨ Click on the Code tab

The code is a single line of code, which replaces the value for the first_name column with its initial:

context.packet.first_name = context.packet.first_name.substring(0,1) + ".";

⇨ Select the Active checkbox

⇨ Click Publish (top)

⇨ Go back to DBGate and refresh the customers table (upper right button)

Note that you now only get the initials for all customers.

Nothing has changed in the database -- we're only transforming the data as it goes from SQL Server to the database client.

This is obviously a simple example -- we could get very fancy here. We could change data depending on any number of factors, we could generate random data, we could hide certain rows and/or columns, we could even insert rows into the result set. We have complete control, and we can make the database behave in ways that would normally be almost impossible, all without any changes to the database or the database client.

A few things to try at this point, if you feel like it:

Hide a row using the remove() method:

if (context.packet.country === 'DK') {
  context.packet.remove();
}

If you deploy this code in the filter and re-run your query, you will see that customer 4 (Daniella Durango) no longer appears in the result set. This type of simple hiding can also be done without code by setting the configuration parameters of the filter, but you get more flexibility with code.


Insert a synthetic row:

if (context.packet.id === 2) {
  let newRow = context.packet.clone();
  newRow.id = -2;
  newRow.first_name = 'Synthetic';
  context.packets.addPacket(newRow);
}

If you deploy this code and re-run your query, you will see a new row appear after customer 2. Again, no changes are made to the database, only to the result set received by the database client.

What have we seen?

In this tutorial, you got a glimpse of how Gallium Data can intercept the traffic between database client and database server, and modify this traffic. This enables you to:

Gallium Data has a number of pre-defined filters, but it also makes it easy to create your own filters and be as sophisticated as you want.

Now, the question is: how will you use it?

What to do next

We encourage you to take Gallium Data for a spin with your own database(s). It's always more interesting to work with your own data than with demo data.

The tutorial project contains several other filters, but they are not active. You can take a look at them and try to activate them:

docker logs -f gallium-data

Gallium Data is free, so you can use it as much as you want, on your machines, servers, in the cloud, wherever.

Consult the documentation for all the gritty details, such as how to use the debugger, or the API for various types of database packets.

Cleanup

Once you're done with this tutorial, and you want to remove everything that was installed,

⇨ Execute the following commands from a command line:

docker stop gallium-mssql
docker stop gallium-dbgate
docker stop gallium-data
docker network rm gallium-net

This will stop all the Docker containers started during this tutorial.

If you also want to remove the Docker images:

docker rmi dbgate/dbgate
docker rmi galliumdata/gallium-data-engine:1.8.5-1867
docker rmi galliumdata/gallium-data-demo-mssql:4

This will remove everything installed by this tutorial.

We'd love to hear from you -- good or bad! Please drop us an email and let us know if you have any questions or comments.


feedback at gallium data dot com

Footnotes

(1) - Gallium Data is just a container, so it runs in anything that can run a container: Docker, podman, Kubernetes, containerd, etc... You can run this tutorial using something other than Docker, but you'll have to translate the command line options. On recent Windows versions, you can run Docker in WSL if you prefer.