Configuration Management for Golang Applications

blog

Typically, our applications involve a number of configurations, such as HTTP/RPC listening ports, logging-related configurations, database configurations, and so on; these configurations can come from a variety of different sources, such as local configuration files, environment variables, command line arguments, or remote configuration centres such as Consul. A configuration can exist in several different sources at the same time and needs to be prioritised. This article describes the use of koanf for configuration management in Golang applications.

viper is a widely used configuration parsing library in Golang programs, but in the process of using it, viper has also exposed many problems.

  • Forcing configuration keys to be lowercase breaks the original semantic definition of TOML, HCL, etc.: forcibly lowercasing keys;
  • Forces the priority of the configuration source: default precedence order;
  • The implementation of configuration parsing such as File, CLI, ENV, etc. is hard-coded in the code and does not provide an API for adding new parsers or customising the parsing process at the application level, making it impossible to extend;
  • Pulling all third-party dependencies at once, even if they don’t use the corresponding configuration sources and parsers, viper still pulls the corresponding dependencies, e.g. ETCD, Consul, gRPC, etc.: Why all the new dependencies?

viper’s codebase implementation is very complex, and it is difficult to understand its design thinking in a short time. viper does not expose extensible semantics to the outside world, and in the process of actual use, if you encounter application scenarios that cannot be covered, you often need to deal with them separately in the business layer, which will add extra intrusiveness to the business logic. Therefore, a more lightweight and easily scalable configuration management implementation is needed.

Overview

knanf is a lightweight and easily extensible Golang configuration library. The v2 version separates external dependencies from the core via a separate Golang Module that can be installed as needed, making it just over a thousand lines of core code.

  • JSON, Yaml and other data formats parsing extensions, just need to implement the Parser interface: koanf/parsers;

    type Parser interface {
    Unmarshal([]byte) (map[string]interface{}, error)
    Marshal(map[string]interface{}) ([]byte, error)
}

  • Data parsing of configuration sources such as Consul, File, Env, etc. requires only the implementation of the Provider interface: koanf/providers;

    type Provider interface {
    ReadBytes() ([]byte, error)
    Read() (map[string]interface{}, error)
}

  • koanf provides Provider and Parser interfaces, by implementing the corresponding interfaces, you can access more configuration parsing methods;

  • koanf itself does not specify the priority of configuration sources, koanf will override existing values in the order in which Providers are called;

The basic operation of koanf can be found in knanf/readme, and will not be repeated in the text, the following content will be introduced on the basis of understanding how to use.

Configuration Priority

koanf overrides existing values in the order in which the Provider is called. So we can achieve the default priority by encapsulating a uniform configuration parsing step. For example, the following code implements a Parse() function, which takes as arguments the configuration path and data format, as well as the corresponding Parser (in the example, local file and Consul are supported).

func Parse(configPath, format string, reader ParserFunc, opts ...OptionFunc) error {
    var parser koanf.Parser

    switch format {
    case ConfigFormatYAML:
        parser = kyaml.Parser()
    case ConfigFormatJSON:
        parser = kjson.Parser()
    default:
        return fmt.Errorf("unsupported config format: %s", format)
    }

    if err := reader(configPath, parser); err != nil {
        return err
    }
}

// OptionFunc is the option function for config.
type OptionFunc func(map[string]any)

// ParserFunc Parse config option func
type ParserFunc func(path string, parser koanf.Parser) error

func ReadFromFile(filePath string, parser koanf.Parser) error {
    if err := k.Load(kfile.Provider(filePath), parser); err != nil {
        // Config file was found but another error was produced
        return fmt.Errorf("error loading config from file [%s]: %w", filePath, err)
    }

    return nil
}

// ReadFromConsul read config from consul with format
func ReadFromConsul(configPath string, parser koanf.Parser) error {
    if err := k.Load(kconsul.Provider(kconsul.Config{}), parser); err != nil {
        return fmt.Errorf("error loading config from consul: %w", err)
    }

    return nil
}

Configuration data can be organised in different data formats and storage methods, and thanks to koanf’s excellent decoupling ideas, we can easily support internal configuration centres

In the case of containerised deployments, we will also modify certain configuration items of the current application based on environment variables, so that dynamic configuration can be performed without modifying the contents of the image.The ENV Provider identifies the environment variables to be resolved by a uniform prefix, and provides an optional callback function that lets the user customise the processing logic for the name of the environment variable. In the example code, our custom callback function removes the uniform prefix and converts all uppercase characters to lowercase.

// Second, read config from environment variables,
// Parse environment variables and merge into the loaded config.
// "PUDDING" is the prefix to filter the env vars by.
// "." is the delimiter used to represent the key hierarchy in env vars
// The (optional, or can be nil) function can be used to transform
// the env var names, for instance, to lowercase them
if err := k.Load(kenv.Provider("PUDDING/", defaultDelim, func(s string) string {
    return strings.ReplaceAll(strings.ToLower(
        strings.TrimPrefix(s, "PUDDING/")), "/", ".")
}), nil); err != nil {
    return fmt.Errorf("error loading config from env: %w", err)
}

CLI Priority

CLI priority is a special scenario that has a default value but can be overridden by command line arguments. Since the koanf repo does not implement configuration parsing for the Golang standard library flag, you need to implement a Provider yourself.

For ease of understanding, only the key parts of the code snippet are provided below. The Golang flagSet provides the VisitAll method, allowing the caller to access all flags set and unset in the flagSet for command-line parameters. Additionally, all flag Value types in the standard library implement the flag.Getter method, which retrieves the current value of the flag and its string representation. By comparing the current value with the initial value, we can determine whether a command-line parameter has been set. Furthermore, koanf also implements the Exist method to check if a key already exists.

  • If the command line arguments are not set and are not set in the low-priority configuration file, then the CLI default values are used;
  • If the command-line parameter is not set, but is set in the low-priority profile, then the value of the low-priority profile is used;
  • If the command line argument is set, then the value of the command line argument is used;
// Read reads the flag variables and returns a nested conf map.
func (p *Flag) Read() (map[string]any, error) {
    mp := make(map[string]any)

    p.flagSet.VisitAll(func(f *flag.Flag) {
        var (
            key   string
            value any
        )

        if p.cb != nil {
            key, value = p.cb(f.Name, f.Value)
        } else {
            // All Value types provided by flag package satisfy the Getter interface
            // if user defined types are used, they must satisfy the Getter interface
            getter, ok := f.Value.(flag.Getter)
            if !ok {
                panic(fmt.Sprintf("flag %s does not implement flag.Getter", f.Name))
            }
            key, value = f.Name, getter.Get()
        }

        // if the key is set, and the flag value is the default value, skip it
        if p.ko.Exists(key) && f.Value.String() == f.DefValue {
            return
        }

        mp[key] = value
    })

    return maps.Unflatten(mp, p.delim), nil
}

Finally the return value of the Read method is a map[string]any, which koanf overrides by merging it with the existing configuration.

Deserialization Configuration

All the configurations of koanf are stored in a global map, but map is an untyped container, which is a loose data structure, not good for doing type checking and data validation. In practice, we often need to deserialise configurations into structures for use in code. koanf provides the Unmarshal method to deserialise configurations into structures. So we need to add mapstructure tag to the structure so that koanf deserialisation can parse the fields correctly.

type BaseConfig struct {
    HostDomain string `json:"host_domain" yaml:"host_domain" mapstructure:"host_domain"`
    GRPCPort int `json:"grpc_port" yaml:"grpc_port" mapstructure:"grpc_port"`
    HTTPPort int `json:"http_port" yaml:"http_port" mapstructure:"http_port"`
    EnableTLS bool `json:"enable_tls" yaml:"enable_tls" mapstructure:"enable_tls"`
}

func UnmarshalToStruct(path string, c any) error {
    if err := k.UnmarshalWithConf(path, c, koanf.UnmarshalConf{Tag: "mapstructure"}); err != nil {
        return fmt.Errorf("failed to unmarshal config: %w", err)
    }

    return nil
}

Summary

The best part of koanf’s design is that it decouples Parser, Provider, and Core, and introduces them in the form of plug-ins, so that each module can be installed separately as needed. The custom implementation of Provider is inexpensive and can be easily extended.