Gallium Data is configured using three main mechanisms:
command line options
options file
repository
A word about files and directory paths
If you run Gallium Data as a Docker container, keep in mind that the files and directory paths you specify must be from the perspective of the Docker container. You will often map directories or files from the host file system into the Docker container. Make sure that the options you specify to Gallium Data correspond to the location of the files or directories as seen from within the Docker container.
Command line options
The following command-line options are accepted by the server.
repository-location=<directory path>
Required. Specifies the location of the repository in which all definitions are stored. The repository is a tree of directories and files, including JSON files, JavaScript files, Markdown files, and a few others. This directory contains everything you do in Gallium Data: projects, connections, filters, etc... It is therefore common to map this directory to your local file system, and to keep it under source control (e.g. Git, PerForce, Subversion, etc...)
backup-location=<directory-path>
Optional but recommended. Specifies a directory in which the repository will be backed up before every publish. This allows you to easily undo any changes, since backups can easily be restored from the Backups page in the admin application.
If this option is specified, Gallium Data will create up to four levels of subdirectories: year, month, day, and time every time the repository is published. Each backup is a complete copy of the repository as it was at that moment in time. Most repositories are relatively small, but over time this can grow and should be pruned from time to time, which can be done from the admin app, or directly in the file system.
meta-repo-location=<directory-path>
Optional. For advanced users only. Specifies where the meta-repository can be found. The meta-repository describes how adapters and filters behave.
rest-port=<number>
Optional. Specifies which port to use to expose the REST API used by the admin app. To use the admin app, you must enable either this option or the rest-port-ssl option (or both). In production, the REST API is usually turned off.
Please note: this specifies the port that will be used in the Docker container. In most cases, you will ask Docker to map this port to a local port, therefore this option is not usually required, since you can map the default port (8080) to any desired local port.
rest-port-ssl=<number>
Optional. Specifies which port to use to expose the REST API (over SSL) used by the admin app. Same note as for the rest-port option.
rest-keystore=<jks-file-path>
Required if rest-port-ssl is specified. This should point to a JKS file containing the private key and associated certificates require to run a TLS service.
rest-keystore-pw=<password>
Required if rest-keystore is specified and points to a JKS file that is password-protected.
web-base=<directory-path>
Optional. For advanced users only, who need to customize the admin app.
rest-password=<password>
Optional. If you want to restrict access to the admin app, you can assign a password using this option, and you will be prompted to log in when starting the admin app. The user name is always "admin".
rest-address-range=<regular-expression>
Optional. If you want to restrict access to the admin app, you can specify a regular expression using this option. Any attempts at using the REST API or the admin app will validate the caller's IP address against this regular expression. This is a Java regular expression.
For instance, if you want to allow only machines on your network, you might use something like: 209\.88\.151\..+
instance-id=<number>
Required only if you run multiple instances of Gallium Data in a cluster with the RedisCluster connector.
node-modules-dir=<directory-path>
Optional. Use the specified directory as the root for Node's require mechanism. In the Docker image, the default value is /galliumdata/node_modules
node-globals=<js-file-path>
Optional. For advanced users only. Points to a JavaScript file containing the global definitions required by installed packages.
node-modules-replacements=<replacements>
Optional. For advanced users only. A comma-separated list of JavaScript package names and their replacements. This is used for JavaScript packages that are not included in the JavaScript engine, such as fs and buffer. For more information, see the GraalVM documentation.
node-modules-download-dir=<directory-path>
Optional. If specified, JavaScript libraries specified in the repository will be downloaded to this directory. By default, the system temporary directory is used. In the Docker image, the default value is /galliumdata/npm_downloads
npm-registry-url=<URL>
Optional. Specifies which registry to use when installing JavaScript libraries, The default is http://registry.npmjs.org/
ivy-settings=<file-path>
Optional. Use this option to change the settings of the Java library subsystem, which allows you, for instance, to use a private Maven repository instead of the default Maven Central. See the Java library page for more details.
If, like most people, you run Gallium Data as a Docker container, these options can be specified using Docker's -e option, but you must then use an underscore instead of a dash in the option name, e.g.:
docker run -d -p 8089:8888 \
-e repository_location=/repo \
-e rest_port=8888 \
-v /users/jdoe/projects/repo:/repo \
galliumdata/gallium-data-engine:<version>
In this example, the REST port in the Docker container will be 8888, and it will be mapped to local port 8089.
Note how /repo is mapped from the host file system. This is common when working on connections and filters because it allows you to access the repository easily, especially for source control.
Options file
All the command-line options can also be specified in a file, which is then passed to Gallium Data as a single command-line option:
settings-location=<path-to-file>
Optional. Should point to a text file containing options as described above.
Again, remember that all the paths specified here are from the perspective of the Docker container.
Example options file
# Required: the location of the repository
repository-location=/galliumdata/repo
# Optional: the location where backups should be created. If not specified, backups are not created.
backup-location=/galliumdata/backups
# Required: the location of the meta-repository.
meta-repo-location=/galliumdata/metarepo
# Optional: the port for the REST/web interface. If neither this nor rest-port-ssl is specified,
# no HTTP service will be started.
rest-port=8043
# Optional: the port for the SSL REST/web interfaces. If neither rest-port nor rest-port-ssl
# is specified, then there will be no REST/web interface, which is common in production.
rest-port-ssl=8044
# If rest-port-ssl is specified, then you should specify where to find the key and certificates
# (and the password for the JKS file, if necessary).
rest-keystore=/galliumdata/KeyStore.jks
rest-keystore-pw=GalliumData
# Optional: where the web files are to be found.
web-base=/galliumdata/web
# Optional: a password required for basic auth for the web and REST interfaces.
#rest-password=Password1
# Optional: a Java regular expression specifying which IP4 or IP6 address(es) are allowed to use
# the web and REST interfaces.
#rest-address-range=(127\.0\.0\.1)|(0:0:0:0:0:0:0:1)
# Optional unless running in a cluster
#instance-id=1
# Optional: the location for Node.js modules
node-modules-dir=/galliumdata/node_modules
# Advanced: where globals are declared for Node modules
#node-globals=/somewhere/globals.js
# Advanced: substitution modules for the ones that aren't built it, e.g. fs
#node-modules-replacements=fs:node-fs-extra,assert:tapsert
# Optional: when installing new JavaScript libraries, the directory in which to download the tarball
node-modules-download-dir=/galliumdata/npm_downloads
If an option is specified in more than one way, the value that will be used is in order of priority:
command-line option
environment variable
options file
In other words, command-line options always trump other options, and environment variables trump the options file.
Repository
The repository is where Gallium Data stores all its definitions: projects, connections, filters, etc... The easiest way to edit it is using the admin app, but (if you are careful) you can also edit the repository directly. This can be handy when doing automated deployments, for instance. Every time you publish in the admin app, the repository is updated (and a backup is created, if the backup option is enabled).
The structure of the repository is relatively straightforward. The top directory contains:
repository.json: describes the repository, including which Java and JavaScript libraries should be made available
projects: contains one directory per project
projects/<project-name>:
projects/<project-name>/project.json : describes the project. The only thing you might edit here is the "active" attribute.
projects/<project-name>/comments.md : description of the project in Markdown format.
projects/<project-name>/crypto: contains private key and certificate-related files for the project.
projects/<project-name>/crypto/crypto.json : can specify the algorithm used for the private key (if present), and the password for the JKS file (if present).
projects/<project-name>/crypto/key.pem : the private key used for SSL/TLS to database clients, and related certificates.
projects/<project-name>/crypto/trust.pem : the certificate(s) that should be trusted from the database server(s).
projects/<project-name>/crypto/keystore.jks : optional - a Java JKS file containing keys and certificates. Creating a JKS file requires a bit of work, but it does provide you with a lot of flexibility.
projects/<project-name>/connections : contains one JSON file per connection in this project.
projects/<project-name>/connections/<connection-name>.json : describes a connection, its type, and parameter values.
projects/<project-name>/connections/<connection-name>.md : a description of the connection, in Markdown format..
projects/<project-name>/connection_filters : contains one directory for each connection filter in the project
projects/<project-name>/connection_filters/<filter-name> :
projects/<project-name>/connection_filters/<filter-name>/connection_filter.json : describes the filter and its parameter values.
projects/<project-name>/connection_filters/<filter-name>/code.js : the JavaScript code for the filter, if any.
projects/<project-name>/connection_filters/<filter-name>/comments.md : a description of the filter, in Markdown format.
The structure is the same for the other three types of filters:
projects/<project-name>/request_filters :
projects/<project-name>/response_filters :
projects/<project-name>/duplex_filters :
Changing the way Gallium Data starts in Docker
By default, Gallium Data starts in its Docker container with the following command, which is in the file RunGalliumData.sh in the root directory of the Gallium Data Docker image:
cd /galliumdata
exec /GraalVM/bin/java \
--module-path /galliumdata/jars \
-Xmx1024M \
-Xshare:off \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.debug=com.galliumdata.engine \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.source=com.galliumdata.engine \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.instrumentation=com.galliumdata.engine \
--add-exports org.graalvm.truffle/com.oracle.truffle.polyglot=com.galliumdata.engine \
--add-opens org.graalvm.truffle/com.oracle.truffle.api.debug=com.galliumdata.engine \
--add-opens org.graalvm.truffle/com.oracle.truffle.api.source=com.galliumdata.engine \
--add-opens org.graalvm.truffle/com.oracle.truffle.api.instrumentation=com.galliumdata.engine \
--add-opens org.graalvm.truffle/com.oracle.truffle.polyglot=com.galliumdata.engine \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.debug=ALL-UNNAMED \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.source=ALL-UNNAMED \
--add-exports org.graalvm.truffle/com.oracle.truffle.api.instrumentation=ALL-UNNAMED \
--add-opens java.desktop/sun.font=ALL-UNNAMED \
--add-opens java.base/java.lang.invoke=ALL-UNNAMED \
--add-opens java.base/java.lang.invoke=retrofit2 \
-Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" \
-Dlog4j.configurationFile=/galliumdata/settings/log4j2.properties \
-Dlog4j.configuration=file:///galliumdata/settings/log4j.properties \
--module com.galliumdata.engine/com.galliumdata.server.Main \
--settings-location=/galliumdata/settings/default.properties
If you need to change this, a good approach is to start an instance of Gallium Data, and extract a copy of that file to your local file system with:
docker cp <container-id>:/RunGalliumData.sh .
then edit that local file to your liking, and finally map it into the Docker container when you start it, e.g.:
docker run -d -v /my/local/directory/RunGalliumData.sh:/RunGalliumData.sh etc...
You can also create your own Docker image to include that file. Create a directory that contains your edited RunGalliumData.sh file and the following file named Dockerfile (replace <version> with the version of Gallium Data):
FROM galliumdata/gallium-data-engine:<version>
COPY RunGalliumData.sh /
You can then run the following command from that directory (replace <your-image-tag> with a name for your new Docker image:
docker build -t <your-image-tag> .
You now have your own customized copy of the Gallium Data Docker image, which you can run like any other.