How should I structure packages for a multiple-binary web application?

[steveivy]

So, newbie to Go here, and I’m working on porting a small web service I wrote in Python to Go. The python version consisted of a single Flask app, with handlers for a REST API that indexed data in Elasticsearch, and a dashboard that provides system status and basic data/search views.

My first attempt in Go sort of replicated that model: one flat package (“main”) written with Gin and elastigo. The package consists of a half-dozen files, and implements the REST api and the dashboard handlers, along with some backend abstractions.

Now I’d like to separate the web service from the dashboard service so that I can work on scaling the rest API, but run the management dashboard by itself, and I’m not sure how to do it. Each binary will need it’s own “main” package, but I want all the code in one repository as there is code that should be shared. Any tips?

[NateFinch]

it’s perfectly fine to put them each in their own subdirectory, so like

github.com/steve/project/restcmd
github.com/steve/project/webcmd

Then put readme etc in the root of the dir.

You can then go get everything in the project by doing `go get github.com/steve/project/… and everything will Just Work™.

[Sevein (Jesús García Crespo)]

In addition to @NateFinch’s comment, I see many projects putting all their main packages together in one directory and it’s frequently called cmd. Two examples off the top of my head are Kubernetes and Camlistore.

Also, I’ve seen other projects mapping all the available commands in one single program, e.g. Hashicorp seems to follow this pattern across all their projects (Consul, Terraform, etc…).

etcd takes a different approach. The root package is a complete program but also its subpackage etcdctl.

[Sevein (Jesús García Crespo)]

Also, a nice read that covers this topic is the article Go: Best Practices for Production Environments written by @peterbourgon (the creator of Go kit).

[steveivy]

Hey @Sevein, I’ve seen that Go in Production article bfore - I’ve got it bookmarked and simply forgot that this is addressed there. I’ll review it again, and thanks @NateFinch as well!

[Makpoc (Bisser Nedkov)]

I had a similar dilemma and went for this suggestion. I don’t know if it’s the best one, but it I think it covers the exact same issue. There was another suggestion later to keep the command line ui in the root of the project, but I have yet to decide whether to do it or not.

[steveivy]

Thanks @Makpoc - another useful perspective. Coming from Python, learning code organization in Go has been a real head-scratcher.

[Sevein (Jesús García Crespo)]

I’ve seen some projects opting for storing all their packages under a pkg subdirectory. I think that’s a good approach if you don’t want to have a mix of Go packages and other things that still you want to manage in the same repo, like docs, the project website, examples, provisioning stuff, integration tests, etc… Camlistore or Kubernetes do this.

But nothing stops you from putting Go packages and other directories together, e.g. that’s what Prometheus, Heka and others do.

[calmh (Jakob Borg)]

This is something we’ve been fighting and iterating on… Current layout on a reasonably sized project looks like this:

Godeps/
  ... lots of dependencies. This would probably be vendor/ if we started again today.
cmd/
  ... lots of binaries / main packages
etc/
  ... various example configs and stuff
gui/
  ... a web app that is compiled into one of the binaries
lib/
  ... our internal packages, some with subdirectories of their own.
  ... this was "internal" at some point, but since this is nowadays enforced and we
  ... actually have a few external uses, it became "lib"
script/
  ... various build supporting Go scripts
build.sh
build.go
README
AUTHORS
... etc standard toplevel stuff

There are a few more top level directories for stuff like graphics assets and so on as well, but not relevant to the Go side of things. So all the Go code lives under cmd/ and lib/, apart from build scripts.

This all builds with standard GOPATH (plus a prepend for Godeps), so internal packages are seen as github.com/organization/project/lib/thepackage.

I’ve been looking into gb as well, but I’m not entirely convinced yet.

[steveivy]

Hey folks, thanks for all the good replies. I tried a structure very much like the one in “Go in Production” article:

HOST:test_service sivy$ tree

.
├── README.md
├── health
│   ├── health.go
│   └── health_test.go
├── test_service_api
│   ├── api_instances.go
│   ├── api_instances_test.go
│   └── main.go
├── test_service_dash
│   ├── dashboard.go
│   ├── dashboard_test.go
│   ├── main.go
│   ├── static
│   │   ├── bootstrap-3.3.2/...
│   │   └── dashboard/...
│   └── templates/...
├── storage
│   ├── storage.go
│   ├── storage_elasticsearch.go
│   ├── storage_elasticsearch_test.go
│   ├── storage_filesystem.go
│   └── storage_filesystem_test.go
└── util_test.go
5 directories, 17 files

Package and imports in storage/storage.go:

package storage

import (
	health "github.com/sivy/test_service/health"
)

type StorageProvider interface {
    // stuff ...
    StatusCheck() health.Healthcheck
}

Running go install, however, gives me:

HOST:test_service sivy$ cd test_api/
HOST:test_api sivy$ go install
# github.com/sivy/test_service/storage
../storage/storage.go:22: undefined: health.Healthcheck
../storage/storage_elasticsearch.go:224: undefined: health.Healthcheck
../storage/storage_filesystem.go:165: undefined: health.Healthcheck
HOST:test_api sivy$

Yeah, I’m confused.

[dlclark (Doug Clark)]

Not 100% sure on the issue here. Does health.go declare its package health?

[steveivy]

Yes, it does - health/health.go:

// health provides datatypes for encapsulating healthcheck data
// for an application.
package health

import (
	"os"
	"os/exec"
	"strings"
)

/*
 Datatypes
*/
type HealthCheck struct {
	Name     string
	Function func() interface{} // preferred
	Command  string             // fallback
}

type HealthCheckResult struct {
	Success bool        `json:"success"`
	Data    interface{} `json:"data"`
}

// Run performs the defined checks for this HealthCheck
func (h HealthCheck) Run() interface{} {
	var checkOutput interface{}
	var err error

	if h.Function != nil {
		checkOutput = h.Function()
	} else if h.Command != "" {
		output, err := exec.Command("sh", "-c", h.Command).Output()
		if err != nil {
			checkOutput = err
		} else {
			checkOutput = string(output)
		}
	}

	// pack up the results as a HealthCheckResult
	var hcr HealthCheckResult
	if err != nil {
		hcr = HealthCheckResult{Success: false, Data: string(err.Error())}
	} else {
		hcr = HealthCheckResult{Success: true, Data: checkOutput}
	}
	return hcr
}

[dlclark (Doug Clark)]

I recreated it locally and figured it out! Check the caps on Healthcheck in the usage vs the HealthCheck type name.

[steveivy]

Oh fer cryin’ out loud.

Thanks @dlclark

[shakeel (Shakeel Mahate)]

Sameer Ajmani has a blog about how to name packages and what should be in your packages, highly recommend reading it. https://blog.golang.org/package-names

[steveivy]

That’s great stuff, thanks! I will be revisiting my code (again!) with this in mind.