A smarter Redis
Everybody loves Redis. It's fast, it's easy to use from almost any programming language, and it works exactly like it should. It's got an impressive list of modules for additional functionality. It's a really good product.
Sometimes, though, you may want Redis to be a little smarter.
Maybe you'd like some clients to have a slightly different view of the data, for instance, or maybe you'd like to change how some of the data is stored or retrieved.
Redis supports scripting using the Lua language. As far as I can tell, though, that is not used very much, I think for a few reasons:
Lua is a fine scripting language, but it's a bit niche (used by less than 1% of programmers according to TIOBE, mostly in games)
scripts have to be explicitly invoked and cannot modify the default behavior of Redis (unlike, say, a trigger in a SQL database)
a script is executed atomically -- all server activities are blocked during its entire runtime, which can be daunting in high-performance environments
So if you want to change how your Redis database behaves, what are your options?
Most people assume that there is nothing more they can do, other than change their Redis clients to do whatever is needed.
Redis is open source, so theoretically we could modify it, but that would require a significant amount of work.
We could also write a custom Redis module, but that is not a trivial task. Modules are written in C, or anything that can produce a C-compatible shared library, and they have to be installed in the server, which is a non-starter in most cloud environments.
Enter the proxy
A much easier option is to put a programmable database proxy between Redis and its clients. This will allow us to modify the requests and the responses as desired. In addition, by having that logic outside of Redis, we can decide which clients go through the proxy (and therefore are submitted to our logic), and which connect directly to Redis.
Does Redis need to be smarter?
Redis does what it does very well, but there are many scenarios in which additional smarts could make a big difference. For instance, you may want to:
integrate Redis into your enterprise authentication mechanism
reject certain requests given certain conditions
change how an existing application queries Redis, without changing the application
add new types of commands to Redis
store some data using your own form of encryption, without affecting the clients
modify certain responses differently depending on who is asking
The list goes on. Let's take a deeper look at a few examples.
Enterprise authentication
Redis has its own authentication mechanism, but it's pretty simplistic. The password(s) can be specified in the redis.conf file, or they can be added at runtime with a command. It's better than nothing, but it does not integrate into well-known authentication systems (the Enterprise edition of Redis does support integration with LDAP, but the open-sourced version does not).
A proxy can add a layer of authentication that will be transparent to the applications by intercepting the AUTH command and performing authentication with a provider such as Active Directory, SAML, OpenID Connect, and so on.
The proxy can simply intercept the AUTH command, and check the credentials with the identity provider. If the credentials are valid, the proxy can then let the client proceed. If the credentials are invalid, the proxy can respond with an error message and (optionally) close the connection.
Even more interesting, the proxy can retrieve the user's profile (things like roles and group memberships) from the identity provider and use that information to determine the user's permissions.
Controlling requests
Redis has the concept of access control lists (ACLs), which allow you to specify what access certain users get to certain objects. This mechanism is nice, though its syntax is not necessarily obvious at first.
For instance, you can specify that user jdoe has permission to execute the GET and MGET commands on keys FOO* and BAR*:
ACL SETUSER jdoe on +GET ~FOO* ~BAR*
ACL SETUSER jdoe on +MGET ~FOO* ~BAR*
Redis 7.0 has extended this mechanism to make it possible to specify which keys are readable, and which are writeable.
This all works fine, but it only goes so far. In particular, Redis' pattern matching is relatively limited, so being able to use a full regular expression engine in the proxy can be an advantage, especially when filtering responses.
In addition, ACLs apply only to keys, and not values. A programmable proxy, for example, can allow you to specify that any object containing an attribute Classification: secret should not be visible to users not satisfying certain conditions.
A programmable proxy can also record all request types during a period of time, then reject any request that has never been seen before -- this is not appropriate for all databases, clearly, but it's a simple and effective security measure.
With a programmable proxy, you can specify arbitrary logic that can accept, reject or modify requests based on any desired criteria, such as the user, the client IP address and location, the nature of the request, the time of day, the day of the week, and so on. Sometimes it's hard to specify exactly what you want using just fixed rules.
Redirecting requests
Because the proxy receives every request before it gets to Redis, it can implement new commands, it can redirect some or all of these requests to a different Redis server, or it can translate them to queries for a completely different database, API, or whatever.
As an example, the proxy can easily intercept requests such as HGETALL ExchangeRates, fetch those values from (say) a REST service, and respond directly to the client, without ever talking to Redis. This allows you to create virtual objects, which do not actually exist in the Redis database, but seem just as real to the clients.
Custom Encryption
Redis has a module called redicrypt that provides new commands to store and retrieve data in encrypted form. This works fine, but it does require you to specifically invoke encryption and decryption from your clients.
A proxy can encrypt and decrypt data transparently as it moves between the clients and the server, using your chosen encryption algorithms and keys, and can encrypt exactly what you need, which may be only certain objects, or certain parts of certain objects. This is all done without any changes to the clients: as far as they're concerned, nothing has changed at all.
Augmenting Pub/Sub
Redis has a simple but effective publish/subscribe model that is easy to use and works quite well in a cluster.
A proxy can of course intercept any subscription commands and redirect them or extend them to other systems like MQ, Kafka, Akka, and so on.
Correspondingly, a proxy can modify, hide or inject messages for clients to receive. This makes it a great way to integrate with other systems.
Renaming commands
Just one more example. It's rather specific, but representative.
Redis allows you to rename commands. If you add the following to your redis.conf file:
rename-command GET cf16263ba56e75f73232248251b9ba70
then no one will be able to execute a GET unless they know the secret word that has replaced the command.
If your Redis server does this, you may have difficulties using some client libraries, since most of them do not support this feature. A proxy can be trivially used to change requests between clients and servers and make this problem disappear.
When is a proxy appropriate?
A programmable database proxy is just a tool, and like any other tool, it can be used well or poorly.
If your database already does everything you need, and you see no point in changing its behavior, congratulations
These are just a few examples. By definition, a programmable database proxy can be programmed to make any changes
I hope to have piqued your interest. A proxy introduces additional complexity to a system, but it also opens whole new possibilities.