Configuration Management in Go
Let’s imagine you have to build a microservice or a web server. Chances are, you’re going to have variables that change between environments in which you run your application.
For instance, while developing you’ll use a local database, and for production an RDS instance. The application has to be configurable, so that you can pass the connection parameters to it without changing any code.
In other words, it’s a good practice to make your applications work in different environments without requiring any code change.
As an alternative scenario, you may build a command line program that behaves differently based on the flags provided by the user. In any case the program has to be configurable.
Although there are many ways to achieve this, in this post I would like to share the methods I use.
Firstly, I wouldn’t use a library for basic configuration management. Even though it saves some amount of code, in the long term it’ll hurt the maintainability of your app/service.
I also prefer creating a config package for this purpose, but you can also do it inside main.go. Putting it inside a separate package makes main.go cleaner and makes testing easier.
Here are 3 possible ways of managing configuration in a Go application.
Environment Variables
The Twelve-Factor App recommends using environment variables.
Let’s implement this approach.
To make things simple let’s say we have only 4 config parameters - database host, port, user and password.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
package config
import (
"os"
)
type Config struct {
DBHost string // Host of database server
DBPort string // ...
DBUser string
DBPassword string
// ...
}
func Parse() Config {
c := Config{
DBHost: getEnvOrDefault("DB_HOST", "localhost"),
DBPort: getEnvOrDefault("DB_PORT", "5432"),
DBUser: getEnvOrDefault("DB_USER", "postgres"),
DBPassword: getEnvOrDefault("DB_PASSWORD", "postgres"),
}
return c
}
func getEnvOrDefault(key string, defaultVal string) string {
v := os.Getenv(key)
if v == "" {
return defaultVal
}
return v
}
|
Here we defined a Config struct that contains the variables and a Parse function. We can define helper functions like getEnvOrDefault to have default values or to parse different types of variables.
And in our main.go we can initialize our config like so:
1
2
3
4
5
6
7
8
|
package main
import "github/mtekmir/go-config-management/config"
func main() {
conf := config.Parse()
// ...
}
|
We can test the config package easily.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
package config_test
import (
"os"
"testing"
"code.com/config"
)
func TestParse(t *testing.T) {
t.Cleanup(func() {
os.Clearenv()
})
os.Setenv("DB_HOST", "hostname")
os.Setenv("DB_PORT", "1234")
os.Setenv("DB_USER", "user")
os.Setenv("DB_PASSWORD", "pass")
c := config.Parse()
if c.DBHost != "hostname" {
t.Errorf("Expected dbHost to be 'hostname'. Got %s", c.DBHost)
}
if c.DBPort != "1234" {
t.Errorf("Expected dbPort to be '1234'. Got %s", c.DBPort)
}
if c.DBUser != "user" {
t.Errorf("Expected dbUser to be 'user'. Got %s", c.DBUser)
}
if c.DBPassword != "pass" {
t.Errorf("Expected dbPassword to be 'pass'. Got %s", c.DBPassword)
}
}
|
I prefer testing packages from the consumer’s perspective. By specifying the package of tests as **_test, we can test using the public API of the package. This way, tests won’t require any change unless the behavior of the package is changed. So we can refactor the internal implementation without changing the tests.
The code is available on Github
Flags
Another way we can configure our application is by using flags. Let’s look at an example.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
package config
import (
"flag"
)
type Config struct {
DBHost string
DBPort string
DBUser string
DBPassword string
// ...
}
func Parse() Config {
dbHost := flag.String("db-host", "localhost", "database host.")
dbPort := flag.String("db-port", "5432", "database port.")
dbUser := flag.String("db-user", "postgres", "database user.")
dbPass := flag.String("db-password", "postgres", "database password.")
flag.Parse()
c := Config{
DBHost: *dbHost,
DBPort: *dbPort,
DBUser: *dbUser,
DBPassword: *dbPass,
}
return c
}
|
Great thing about using flags is that it provides documentation through usage arguments. Plus we can provide default values without writing helper functions.
We can run our program with -h flag to understand what the flags are for. It’ll output the usages.
1
2
3
4
5
6
7
8
9
|
Usage of /tmp/go-build1347142465/b001/exe/main:
-db-host string
database host. (default "localhost")
-db-password string
database password. (default "postgres")
-db-port string
database port. (default "5432")
-db-user string
database user. (default "postgres")
|
Also, we can supply the flags with a run command that uses environment variables if we need to.
go run main.go -DATABASE_URL $DATABASE_URL -ORDER_SVC_ADDR $ORDER_SVC_ADDR -DATABASE_MAX_IDLE_CONNECTIONS $DATABASE_MAX_IDLE_CONNECTIONS
We can test it like so:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
package config_test
import (
"code.com/config"
"os"
"testing"
)
func TestParse(t *testing.T) {
os.Args[1] = "-db-host=hostname"
os.Args[2] = "-db-port=1234"
os.Args[3] = "-db-user=user"
os.Args[4] = "-db-password=pass"
c := config.Parse()
if c.DBHost != "hostname" {
t.Errorf("Expected dbHost to be 'hostname'. Got %s", c.DBHost)
}
if c.DBPort != "1234" {
t.Errorf("Expected dbPort to be '1234'. Got %s", c.DBPort)
}
if c.DBUser != "user" {
t.Errorf("Expected dbUser to be 'user'. Got %s", c.DBUser)
}
if c.DBPassword != "pass" {
t.Errorf("Expected dbPassword to be 'pass'. Got %s", c.DBPassword)
}
}
|
Using flags would be the go-to method for cli applications.
The code is available on Github
Configuration Files
Another method would be to read the config from a file. The file can be in any format - json, yaml, toml etc. One thing to note here is that, if the file contains secrets, we shouldn’t check it to a git repo.
Let’s say we have a server that runs multiple cron jobs. We can use json files to configure these.
We can store the config of cron jobs in a separate struct, then parse the configuration parameters from the json file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
package config
import (
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"os"
)
type Config struct {
// ...
CronConfigs CronConfigs
}
type CronConfigs struct {
InventoryCron CronConfig `json:"inventoryCron"`
InvoicesCron CronConfig `json:"invoicesCron"`
}
type CronConfig struct {
Schedule string `json:"schedule"`
Description string `json:"desc"`
Disabled bool `json:"disabled"`
NotifyEmail []string `json:"notifyEmail"`
}
func Parse() (Config, error) {
// ...
cronConfPath := flag.String("cron-config-file", "cron_config.json", "path of cron config file")
flag.Parse()
file, err := os.Open(*cronConfPath)
if err != nil {
return Config{}, fmt.Errorf("failed to open config file. %v", err)
}
bb, err := ioutil.ReadAll(file)
if err != nil {
return Config{}, fmt.Errorf("failed to read config file. %v", err)
}
var cc CronConfigs
if err := json.Unmarshal(bb, &cc); err != nil {
return Config{}, fmt.Errorf("failed to unmarshal config file. %v", err)
}
conf := Config{
CronConfigs: cc,
}
return conf, nil
}
|
Let’s make the config file path configurable so that we can specify a different one if we need to. It will then open the file and decode the contents.
And in our main.go we can initialize our config just like before.
1
2
3
4
5
6
7
8
9
10
11
12
|
package main
import (
"code.com/config"
)
func main() {
conf, err := config.Parse()
if err != nil {
// ...
}
}
|
We can test this by creating a config file in config/testdata directory.
1
2
3
4
5
6
7
8
9
10
|
.
├── config
│ ├── config.go
│ ├── config_test.go
│ └── testdata
│ └── cron_config.test.json
├── cron_config.json
├── go.mod
├── go.sum
└── main.go
|
Now in our test we can pass the test config file path with -cron-config-file flag.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
package config_test
import (
"os"
"testing"
"code.com/config"
"github.com/google/go-cmp/cmp"
)
func TestParse(t *testing.T) {
expectedConf := config.Config{
CronConfigs: config.CronConfigs{
InventoryCron: config.CronConfig{
Schedule: "30 0 * * *",
Description: "Cron to calculate inventory stats",
Disabled: false,
NotifyEmail: []string{"[email protected]"},
},
InvoicesCron: config.CronConfig{
Schedule: "10 0 * * *",
Description: "Cron to generate invoices",
Disabled: true,
},
},
}
os.Args[1] = "-cron-config-file=testdata/cron_config.test.json"
c, err := config.Parse()
if err != nil {
t.Fatalf("failed to parse config. %v", err)
}
if diff := cmp.Diff(expectedConf, c); diff != "" {
t.Errorf("Configs are different (-want +got):\n%s", diff)
}
}
|
I use go-cmp for comparing complex types in tests. It displays diffs more clearly compared to testify/assert in my opinion.
The code is available on Github
I usually end up using environment variables for app config. In the end it depends on your or your team’s preferences. For example, Kubernetes allows ConfigMaps and Secrets to be used inside a container command, as environment variables or as a file to be read by the application, so we can just use any one of these methods.
If it’s not a toy project, I handle configuration inside a config package and include a basic test. I really like to keep main.go as clean as possible.
I hope this was helpful. If you find any mistakes or have any remarks, please let me know by dropping a comment. Any feedback would be appreciated.
