Not sure how to structure your Go web application?
My new book guides you through the start-to-finish build of a real world web application in Go — covering topics like how to structure your code, manage dependencies, create dynamic database-driven pages, and how to authenticate and authorize users securely.
In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way.
This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial.
Installing a driver
First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at https://redis.io/clients#go.
The key differences are that Redigo is completely self-contained (with no external dependencies) and it has a smaller, simpler API than Radix. Radix, on the other hand, provides support for Redis sentinel and cluster implementations.
Throughout this post we'll be using the Redigo driver.
Getting started with Redis and Go
As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis.
There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value.
So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of:
To do the same thing from a Go application, we need to combine a couple of functions from the gomodule/redigo/redis package.
The first is the Dial() function, which returns a new connection to our Redis server.
The second is the Do() method, which sends a command to our Redis server across the connection. This returns the reply from Redis as an interface{} type, along with any error if applicable.
Using them is quite straightforward in practice:
In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything except check the return value from Do() for any errors.
Working with replies
When we are interested in the reply from Redis, the gomodule/redigo/redis package contains some useful helper functions for converting the reply (which has the type interface{}) into a Go type we can easily work with. These are:
redis.Bool() – converts a single reply to a bool
redis.Bytes() – converts a single reply to a byte slice ([]byte)
redis.Float64() – converts a single reply to a float64
redis.Int() – converts a single reply to a int
redis.String() – converts a single reply to a string
redis.Values() – converts an array reply to an slice of individual replies
redis.Strings() – converts an array reply to an slice of strings ([]string)
redis.ByteSlices() – converts an array reply to an slice of byte slices ([][]byte)
redis.StringMap() – converts an array of strings (alternating key, value) into a map[string]string. Useful for HGETALL etc
Let's use some of these in conjunction with the HGET command to retrieve information from one of the album hashes:
It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command, or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a float64). There's no way of knowing which kind of error it is unless we examine the error message.
If you run the code above you should get output which looks like:
Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct.
Running this code should give an output like:
Or an alternative, and arguably neater, approach is to use the redis.Values() and redis.ScanStruct() functions to automatically unpack the data to the Album struct, like so:
Using in a web application
One important thing to know about gomodule/redigo/redis is that the Conn object (which is returned by the Dial() function we've been using so far) is not safe for concurrent use.
If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use establish a pool of Redis connections, and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool.
We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions:
Method
Path
Function
GET
/album?id=1
Show details of a specific album (using the id provided in the query string)
POST
/like
Add a new like for a specific album (using the id provided in the request body)
GET
/popular
List the top 3 most liked albums in order
To avoid detracting from the main purpose of this blog post (which is talking about Redis) we'll use a deliberately over-simplified pattern for our web application. If you'd like to follow along, create a basic application scaffold like so…
…And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run:
In the albums.go file we'll define a global variable to hold a Redis connection pool, and we'll re-purpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers.
Alright, let's head over to the main.go file. In this we will initialize the connection pool and set up a simple web server and HTTP handler for the GET /album route.
It's worth elaborating on the redis.Pool settings. In the above code we specify a MaxIdle size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly. The IdleTimeout setting is set to 240 seconds, which means that any connections that are idle for longer than that will be removed from the pool.
If you run the application:
And make a request for one of the albums using cURL you should get a response like this:
Using transactions
The second route, POST /likes, is quite interesting.
When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set.
This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for race conditions to occur.
The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group).
Let's create a new IncrementLikes() function in the albums.go file which uses this technique.
We'll also update the main.go file to add an addLike() handler for the route:
If you make a POST request to like one of the albums you should now get a response like:
Using the Watch command
OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the albums.go file. In this function we need to:
Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set.
Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice.
Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data.
The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client or connection modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful.
Using this from our web application is nice and straightforward:
One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed.
Making a request to the GET /popular route should now yield a response similar to:
If you enjoyed this article, you might like to check out my recommended tutorials list or check out my books Let's Go and Let's Go Further, which teach you everything you need to know about how to build professional production-ready web applications and APIs with Go.