Dotconfig - a simple package to help with (micro)service configuration

Releases
1

This was my Saturday project. I mostly built this for internal use because I found myself copying/pasting similar code across multiple projects, but figured somebody else may be able to get some utility from it:

These days I mostly am deploying my Go code to cloud providers (Google Cloud, AWS, and Azure) and exposing configuration settings via secret managers (aren’t we all?). Sometimes I want to run these services locally and inject environment variables via key/value pairs. This package wraps reading the key/value pairs and uses reflection to turn environment variables into structs:

package main

import  "github.com/DeanPDX/dotconfig"

// Our contrived AppConfig with env struct tags:
type AppConfig struct {
        MaxBytesPerRequest int     `env:"MAX_BYTES_PER_REQUEST"`
        APIVersion         float64 `env:"API_VERSION"`
        IsDev              bool    `env:"IS_DEV"`
        StripeSecret       string  `env:"STRIPE_SECRET"`
        WelcomeMessage     string  `env:"WELCOME_MESSAGE"`
}

func Main() {
        config, err := dotconfig.FromFileName[AppConfig](".env")
        // config is ready to use. Float/bool/int values will be 
        // correctly parsed. If you are developing locally you can
        // put an .env file in your working dir to supply the values.
        // In the cloud they should come from a secret manager.
}

My goal here was to keep is as simple as possible. Define your struct with env tags that map properties to environment variables. Then it “just works” in local dev (either by setting local env variables or including a .env in the working dir) and in the cloud where environment variables are supplied via secret managers.

Here’s the expected format for .env files:

MAX_BYTES_PER_REQUEST='1024'
API_VERSION='1.19'
# All of these are valie for booleans:
# 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False
IS_DEV='1'
STRIPE_SECRET='sk_test_insertkeyhere'
# Right now supporting newlines via "\n" in strings:
WELCOME_MESSAGE='Hello,\nWelcome to the app!\n-The App Dev Team'

The decoding logic borrows heavily from encoding/json in the stdlib. I also exposed a function called FromReader in the event that you want to set key/value pairs from something that isn’t a file or you want to manage file IO yourself.

If you have any questions feel free to ask. I’m also open to feedback. Again - I wanted to keep this as simple as possible so the entire package right now is under 200 lines of code. More info and runnable example in the godoc.

1 Like
2

Looks great. and you are correct I also needed to code something like this but more specific of cousrse.

are you sure that the single quotes are usual in .env files?

STRIPE_SECRET=‘sk_test_insertkeyhere’
vs
STRIPE_SECRET=sk_test_insertkeyhere

can you make it work for both cases?

Should be here:

// Turn a line into key/value pair. Example line:
		// STRIPE_SECRET_KEY='sk_test_asDF!'
		key := line[0:strings.Index(line, "='")]
		value := line[len(key)+2:]
		// Trim closing single quote
		value = strings.TrimSuffix(value, "'")
		// Turn \n into newlines
		value = strings.ReplaceAll(value, `\n`, "\n")
		os.Setenv(key, value)

For errors / info I would need the following:

  1. if the .env file has an unused line.
  2. if a needed entry is blank or missing.

(just not here)
STRIPE_SECRET=
STRIPE_SECRET=‘’

  1. if there is a typo or something is mispelled wrong line.

then the user can decide, based on errors, if they abort the app at init and force the user to enter a specific field if this is needed.

1 Like
3

Karl, I appreciate you taking the time to give detailed feedback! Thank you! I’ll make some GitHub issues to track this stuff.

Good point. .env files aren’t really a standard and it looks like maybe quotes for values are optional. Take a look at this node implementation:

I guess I should make them optional (I was mostly adding them for future support of newlines with “\n” in the string). And I should also allow both single and double like that library does

You mean .env has STRIPE_SECRET but there’s no corresponding property on your config struct? Interesting. I’m not sure if I would usually want an error there. But I can also see how you don’t want old config settings hanging around that are deprecated. I might take a pass at this as an option param.

Yeah - this is getting into validation and I’m of two minds about validation. I could easily add a required struct tag and make sure something is there. But - usually validation is more complicated than that. So I usually have my config struct validate itself like so:

type AppConfig struct {
	IsDev        bool   `env:"IS_DEV"`
	StripeSecret string `env:"STRIPE_SECRET"`
	//...
}

func (c *AppConfig) Validate() error {
	// Contrived but my point is: I often validate things differently
	// when I know it's a local dev environment so I don't need to be
	// as strict.
	if !c.IsDev && len(c.StripeSecret) < 128 {
		return errors.New("must provide secure stripe secret")
	}
	return nil
}

That said, I will think about how/what to implement in terms of what you’re describing for required. That should be simple enough. Often times in my config structs I try to think about the zero value and, if I can, make them useful. IsDev is actually one such case where failing to set it to “true” means by default I’m running in a production-like mode where I am going to be strict. You have to opt in to the more relaxed “dev” mode, which nobody would accidentally do in a production-like environment (I hope!).

Are you trying to get at fuzzy matching here? Like err: STRIPE_SECRETT was found in .env file. Did you mean STRIPE_SECRET?. Idea being I find un-matched env var names and if there’s a similar struct tag, recommend it? I’ll think about that more too. :slight_smile:

Anyway, the good news is: all of my functions accept variadic options elements, so I can be flexible adding new stuff like this without breaking any of my services that depend on this.

4

hm, I can’t remember what I meant exactly.

For starters, I think the dotcofing should just look to see if there is any discrepancy between the .env and the structure. so no missing fields or too many fields. on one side or the other.

It should also collect information/errors and return them as an array so that if there are 2 errors, the user knows immediately what to change.

I think I can help with that next week (busy until then) and make a request on gh.

5

I left it kind of loose in terms of validation partially on purpose because I envisioned a scenario where you want to mix and match. Like some env vars are coming from secret manager / whatever but you also have an .env file to override some options. Anyway, I think we can achieve both your “strict validation” scenario and my “loose validation” one with options.

6

I made a gh pull request. the main change is the single quote thing and the errorCollection instead of a single err.

If you like it you can maybe refine it a bit and then I can use it in go4lage :slight_smile:

7

I’ll take a look. Thanks, Karl!