Comparing the structure of web applications

[marcusolsson]

This could probably have been a blog post but I figured that I might try posting it here this time.

There was a thread recently that discussed examples of web applications. As far as I know, there is no preferred way of structuring your web application, although some variants might be more idiomatic than others. A couple of examples came up and I started checking them out to look for similarities. I believe that one of the strengths of Go is that you can choose whatever way that solves the particular problem you are having. Still though, it might be fun to see what people tend to go with

Starting out from scratch, your service will most likely fit into the main.go. Handlers, middleware and all.

main.go

As your application starts exposing multiple endpoints, you might at some point want to extract your endpoints to separate *.go-files. Either you create a single handlers.go or you have one file for each resource:

groups.go
main.go
users.go

Note that this will make go run main.go fail and you might instead use the go build && ./app approach.

Now, many (if not all) projects I have looked at prefer to move the resource files into a server (or web, or app) directory.

main.go
server/
	groups.go
	users.go

At this point, I would like to point out some things that are common enough to consider general practice.

Persistence

At this point you might want to start persisting your posts, users or what you may have. Almost every application I have looked at will create a root-level package that contains your storage implementation.

database, db, store, storage.

Configuration

Applications that need additional configuration will have a root-level directory containing json, yaml, xml or even *.go files containing structs that are initialized from environment variables in the package func init().

configuration, config, conf.

Templates

Templates will also be located at root-level in its own package.

templates, tmpl.

Things that are common enough to be mentioned are:

Utilities

Interestingly enough, utility seems to be popular as it exists in many of the popular applications I have looked at.

util, utils

Models

Some of the apps will also have the model package containing their domain objects.

model, models

Evolving further

So let’s say we need a database, configuration as well as templates, our project will look something like this:

configuration/
database/
main.go
server/
	groups.go
	users.go
templates/

Now, some of the applications provide a command-line interface or similar applications. Let’s create a cli package at root-level.

cli/
	main.go
configuration/
database/
main.go
server/
templates/

By keeping a main.go in the root-level, to me this signals that there is a notion of a primary application. Is it our cli or the server? Some projects move the root-level main.go into server while others create a cmd/ directory containing the supporting applications.

cmd/
	cli/
		main.go
configuration/
database/
server/
main.go
templates/

or …

cli/
	main.go
configuration/
database/
server/
	main.go
templates/

If we look closer at the server package, several of the applications create a middleware.go and/or router.go to extract functionality from main.go. You might even find apps that create separate packages within server/.

cli/
	main.go
configuration/
database/
server/
	middleware/
	router.go
	main.go
templates/

Obviously, not all web applications I have looked at look the same but by normalizing package names and squinting a bit, there seems to be some common ways to structure web applications in Go.

If you would like to check out the web applications for yourself, these are some of the examples I have used when doing this post. Note that some of these are other permutations of the ones I have listed above.

• EmileVauge/traefik
• Unknwon/gowalker
• drone/drone
• fiorix/freegeoip
• gogits/gogs
• golang/blog
• golang/gddo
• kabukki/journey
• mattermost/platform
• jmoiron/monet

These are the ones I managed to find (as well as a few smaller applications that fit into some of these “archetypes”.

Do you agree with my findings? Do you know of any other applications that won’t fit into any of these examples? I would be happy to update this post based on your observations.

[Egon (Egon Elbre)]

I’ve expressed my dislike for the “models”, “view” etc. folders before.

tl;dr; organize your code by value, not by their form.

What if your project requires 500 models, 1000 views? Are you going to stick them all under those folders. How usable would be standard library if it were organized similarly.

Idiomatic Go has packages created by value that they provide – not according to the engineering details.

Create the folder structure such that just by looking at the folders you can figure out what the program does.

For example, when you are writing an issue tracker, make the packages “issue”, “tracker”, “webclient” – because those are the most important things in a issue tracker. “Issues” are the central concept, “tracker” provides functionality to track things, “client” provides a way for users to use the system.

Now you deconstruct each of those artifacts similarly. How does “tracker” provide it’s value – i.e. it must have an “User” and probably some “Service”, hence you get… “tracker.User” and “tracker.Service”… and similarly, “issue.Info”, “issue.Status”, “webclient.Server”.

While developing you will adjust these folders as necessary, merge or split, depending how your knowledge about the “value provided” increases.

This approach will look idiomatic. Already all the names will look better, you won’t have “model.IssueStatus”, but rather “issue.Status”.

You can look at my extended comments here https://groups.google.com/d/msg/golang-nuts/1z5kL5FCDdU/Zq6l0jUzAwAJ

As for real-world example I can share: https://github.com/raintreeinc/knowledgebase/.

I will soon write a longer post on this and examples how to write projects and architect the code well from start.

[marcusolsson]

I agree with you 100%. This is actually part of the reason I wanted to the “survey”. Somehow, when designing libraries, most seem to agree the packages should be about what they provide as opposed to what they contain. When looking at web applications though, this does not seem to be the norm.

Several of the applications I’ve found are very similar to how Rails applications are structured. I actually came across this article where the author admits it to be a “compromise between idiomatic Go and ruby on rails”. I really would like to see more web applications where we don’t do compromises.

I will take a closer look at your knowledgebase example but at first look it does indeed look more like what I have in mind.

[kennygrant]

This is an interesting question, though I don’t believe we should be prescriptive about how web applications are structured - the best structure might vary wildly depending on the app ( an API, a document server, a web app, client & server, multiple services etc). So there will be no one perfect structure for web apps under go. I do think they should be go-gettable, easily bootstrapped if they use a database, use godoc to generate great docs, and structure their apps using packages for bits of functionality that might be shared between web apps, so no ‘models’ or ‘controller’ packages. That means others can easily go get them and get started, and pick out bits of functionality (say auth, currency formatters, or even a comments pkg say) to use it elsewhere.

For what it’s worth, I tend to take a very different approach from the apps you have listed above - a more structured approach based on the urls used to access the application, which hopefully leads to easy discovery, even on much larger apps with 10s of resources. Something like this:

server.go - entrypoint with main(), includes other packages
bin - for binaries
db - for backups, migrations etc
log - for logs
public - the public dir of the website, static resources, compiled resources
secrets - config
src - a set of packages split up by resource, 3rd party and util packages in lib
  app - app setup, routes, config, db setup, shared layouts etc
  lib - packages for auth, editing, payments, email, so mail.Send etc
  users - handles urls under /users
     views - templates for users resource
     assets - js,css for users compiled from here into public
     actions - handlers for users resource 
     users.go - user model
     users_test.go - tests for user model
  pages 
  etc...

This forces you to isolate resources to a greater extent than a shared ‘models’ pkg, and keeps all the functionality, code and resources to do with a given resource in your web app together (like pages, posts, stories, users, etc). Models know nothing about other models. Handlers are in a separate sub-package to let them refer to other models/resources where necessary (for example users actions might join posts or something). Having used rails extensively in the past, a few things I feel it got wrong which should not spread to go:

• Templates and assets like js/css are in a separate folder hierarchy from the views/code they interact with - this is more painful the larger the app gets (same problem with a separate templates folder in go).
• Controllers are bags of actions, which should often be completely independent unrelated handler functions - there is no real need for a ‘controller’ at all, and many rails controllers are almost empty - if so why do they exist? Also, inheritance.
• The framework attempts to do too much behind the scenes (aka magic)
• Addiction to using an ecosystem of gems extending models and rails itself which means many apps glue together too many disparate libraries leading to fragility and dependency hell - this is a danger in go apps as well that try to go down the path of gluing together lots of disparate libraries - I like to vendor most pkgs in lib instead to be clear about dependencies and the keep the number low.

However it did get a lot right (primarily the helpful command line tool I believe), and having a common structure to apps does help you get up to speed quickly in someone else’s app (or even your own if you come back after 6 months), so there is some value in that. It would be really interesting to hear how others have structured their apps and why.

[Egon (Egon Elbre)]

Note that while writing Knowledge Base, it was a discovery process also, so some of the things aren’t as perfect as they could be.

Also, I’m not sure whether there should be an echo structure for the client or not. I.e. whether to have a separate “client” folder for web sites or to have them mixed with the relevant “domain” package structure. Separate “client” folder feels more right, but I have to experiment and research it whether it really is always the best way. Echoes are part of wholeness according to Christopher Alexander, so it just might be the right thing to do. Why is this relevant? Notice that “strong centers” == “package by value”, “levels of scale” == “recursive decomposition of value” and “boundaries” == “packages”. I suspect that in some cases it makes sense to mix them together and in some other not… I haven’t figured out all of the patterns yet.

Also here’s the latest work in process, where I’m trying out these ideas:

wiki
│   page.go <-- contains info about the page
│   user.go <-- contains info for the user
│   history.go <-- handles page history
│   server.go <-- integrates the pieces
│   db.go <-- interface for data storage
├───assets <-- common assets for editor and view
├───auth <-- handling authentication
├───editor <-- editor for the wiki pages
│   │   index.html <-- requests the assets and js
│   │   editor.js <-- main javascript
│   │   server.go <-- serves editor
│   └───api <-- api for the server
└───view <-- html server for the wiki pages
        server.go

The whole things is layed out according to this principle. This “wiki” is valuable because it has pages, has a convenient editor and provides a way to view the pages. Editor serves it’s own content, but also contains an api etc. One thing I haven’t figure out is where to put the “front door”, i.e. “main.go”. I’m not convinced of the “cmd” sub-folder yet, but as a front-door it seems to fit the description. This structure feels very right to me, there’s very little waste in the structure and everything is condensed down. Of course there are a lot of pieces missing.

As a general statement, I’m pretty confident that the “package according to value” gives better architecture and results than alternatives. It keeps things more alive, generates better understanding about what is being built and gives life to more ideas and value.

[marcusolsson]

@kennygrant

So there will be no one perfect structure for web apps under go.

I’d like to stress this as well. The apps I’ve listed are no doubt great apps and each structure will most likely solve the problem they were having.

@kennygrant

For what it’s worth, I tend to take a very different approach from the apps you have listed above

Thanks for sharing your application! It is indeed different from what I have found. I like the way the users and pages resources are packages. It feels closer to the kind of vertical partitioning that I’m looking for. Personally though, I would probably move the contents of src to the root. This way, I think, you would get a better overview of what the app actually does.

[marcusolsson]

This feels like a very clean layout. I’ve been thinking that domain objects/logic should gravitate towards root simply because, well, they are the application. Much like the page, user and history (I’m guessing) in your example.

Out of curiosity though, do you mind showing what kind of API does the editor expose?

Personally, I like having an obvious entrypoint, like a single root-level main.go that gives me a rough overview of the application before digging deeper. For a single-binary application, I rarely use the cmd directory. I tend to use cmd for supporting applications that aren’t worth extracting to their own repository.

[Egon (Egon Elbre)]

@marcusolsson

Out of curiosity though, do you mind showing what kind of API does the editor expose?

It’s nothing fancy, just a REST create, delete, update, fork, move, load, history, list (of all pages), listen (websocket handler for live-updates); and probably some will be added later. It’s more of a “editor needs to use an API” and not “editor exposes an API”. As for the program side, there is “editor.Server”, that will use “api.Server”.

@marcusolsson

I’ve been thinking that domain objects/logic should gravitate towards root simply because, well, they are the application.

What if you have 2000 domain objects?

That’s why I was very careful in my language. It’s not “put the domain objects at root”… it’s about figuring out what is the “primary value” in software, then create namespaces, functions, types for them, and then you do the same for each value part.

“editor/api” is a very important part for the “editor”, but it’s not the valuable part when looking at the whole application. Similarly you will have some domain objects that may not belong at the top-level.

It’s not just about “domain” objects either, i.e. the “editor” is not a domain object, by any means… however it is a very valuable part of the whole application, hence it’s kept at the top-level.

Sometimes it’s not even a thing that is important, but rather interaction between things. For an imagined and trivialized bank one could create a package named transfer with a method transfer.Money(from Source, to Sink, amount Money), with interfaces transfer.Source and transfer.Sink, because it’s one of the important value points that the Bank provides.

If it’s a highly technical subject, there might not even be a “domain” object per-se… for example take a look at the standard packages. It’s value comes from providing commonly used, needed for compiler and hard to get right packages.

The “deconstruct by value” is the common thread I’ve found among code structures. I.e. it is not only on package level, it also works inside the package and when structuring types. e.g. the wiki.Server will contain assets.Server, auth.Providers, editor.Server and view.Server and DB. When you take a look at the method level, you see something similar – if you only have things that mention the important things for that function – it becomes clear; you won’t have mixing of abstraction levels problem.

[marcusolsson]

I used the word “gravitate” here as opposed to “move” because, as you say, there are other forces that determine choice of structure. In your wiki example, I’m able to get a feel for what the application provides since the files and directories at root-level describe core concepts.

I have been porting a Java application to Go to explore these ideas. There is still some Java heritage and I’ve yet to clean it up since switching to go-kit but it should be able to serve as an example.

github.com/marcusolsson/goddd

Here, as you say, a couple of the packages are not things per-se. booking and routing are use-cases rather than actual domain objects. A bonus being that this makes better use of the package names such as booking.Service and voyage.Number.

The reason I was asking about the API is because I’m thinking of moving handlers and corresponding middleware into the packages rather than to have them at root. I’d very much like to hear your opinion if you get to opportunity to check it out.

[Egon (Egon Elbre)]

First, I must say, it’s quite nice to read that code - it feels nice. But, since you asked for critique:

https://github.com/marcusolsson/goddd/blob/master/voyage/voyage.go#L14

Here I would name it voyage.Info instead of voyage.Voyage… although I don’t feel very strongly about it. I dislike repetition.

https://github.com/marcusolsson/goddd/blob/master/booking/booking.go#L31

Here I would call the fields cargos, locations and router, since the *Repository and *Service part really don’t add any clarity.

https://github.com/marcusolsson/goddd/blob/master/location/location.go#L10
Here I would name it location.UN or location.Code.

https://github.com/marcusolsson/goddd/blob/master/repository/repositories.go#L9
I would rename repository.cargoRepository → repository.cargo

Here: dddsample-core/src/main/java/se/citerus/dddsample/domain/model at master · citerus/dddsample-core · GitHub. It does the right decomposition, i.e. each of those pieces corresponds to important domain knowledge, hence it should also be reflected in Go code, separate them. While looking at the cargo package, it felt confused, what value it is providing. I believe splitting it would create some circular dependencies because of functions, such as https://github.com/marcusolsson/goddd/blob/master/cargo/cargo.go#L39, but shouldn’t be a method on cargo.Info in the first place… I suspect there should additionally be two packages delivery and route (rename routing.Service → route.Finder).

I’m not convinced about the inspection.Service and handling.Service, they don’t feel right for some reason. But, I don’t have a “confident better approach” either. I would maybe try event-sourcing (e.g. https://github.com/egonelbre/event/tree/master/example/guestlist). I suspect there might be some other approaches as well.

@marcusolsson

The reason I was asking about the API is because I’m thinking of moving handlers and corresponding middleware into the packages rather than to have them at root.

For middleware, yes a separate package middleware that contains sub-packages, such as middleware/logging sound good. Use the middleware as an organizing folder, rather than a package.

For handlers, it depends how complicated things get. I think the echoing of the main structure might be just right. I.e. similarly to the middleware/logging you would have api/bookingapi… alternatively it would also make sense to make booking/bookingapi… I’m don’t have a good opinion about this, yet.

[marcusolsson]

Whoa, that is some great feedback If you don’t mind, I will DM you with my reply to keep this thread from deviating to far from the original post. Thanks!

[Egon (Egon Elbre)]

@marcusolsson

If you don’t mind, I will DM you with my reply to keep this thread

Sure, I don’t mind… although I think these conversations are quite relevant.

Also noticed one additional thing location.ErrUnknownLocation → location.ErrUnknown… maybe make a tool that checks for such stutters in naming :), I suspect there are more that I missed.

[kennygrant]

@marcusolsson

Personally though, I would probably move the contents of src to the root. This way, I think, you would get a better overview of what the app actually does.

Yes, this is a nice idea. I had them in a subdir so that I can scan that dir for assets (for asset minification on deploy), and for templates, but could probably move them up a level and scan the entire app directory for those (would have to ignore assets already in public though). I wasn’t particularly happy using ‘src’ as a name anyway, and the fewer levels the better, so I’ll look into this.

[marcusolsson]

Alright. In that case I will post my reply here after all : )

Since you’re mentioned event sourcing, I’m going to assume that you are familiar with DDD.

• voyage.Voyage is one of the aggregate roots in the original application and since domain experts will tell you that “a voyage will have a voyage number and a schedule”. Info would not convey the same meaning.
• Same thing regarding the UN Locode which is also part of the domain language.
• Good point about repository naming. I’ve tried to remove as much repetition as possible but this one manage to hide in broad daylight.
• The cargo package should indeed be split. Problem is that, as you have noticed, the Java version has a circular dependency which prohibits having for example handling.Event and handling.History instead of cargo.HandlingHistory.

My initial thought was to keep the packages “pure” and not to include infrastructure code, but I’m warming up to the thought of having each packages expose the corresponding endpoints. proxying.go for example, is middleware specific to the routing package but on the other hand, it feels wrong to have a middleware package for every booking, routing and so on.

As a sidenote I’m actually working at the company that did the original Java application and we are currently thinking about how the improve the actual domain model as well. The Go port however, is something I’m doing as a personal discovery process. Event-sourcing have indeed been something we’ve thought about. In fact, there already is a C# port of the application that has been rewritten using CQRS/ES.

[Egon (Egon Elbre)]

@marcusolsson

Voyage is one of the aggregate roots…

voyage.Aggregate? But sure, voyage.Voyage is also fine.

@marcusolsson

Same thing regarding the UN Locode

Whan domain experts speak, what do they use? “location code”, “locode” or “UN Locode”? So, indeed that should be used… but I suspect "

location code" or “locode” is used when talking.

@marcusolsson

circular dependency which prohibits

The circular dependency comes from methods, not from the structures themselves – this suggest that the methods should be funcs instead. I didn’t notice any circular dependency in data. First the cargo.TrackingID → cargo/tracking/id.go. Then remove all methods from cargo types, split the packages… and either add methods to their appropriate packages or create a new package for them.

Those methods seem more like interactions or use cases, rather than methods.

[fazal]

I also do not like to make views/ or model/ folders, it just makes things very generic and meaningless. Though I have a similar structure as OP.

For example I had a forum, I’d structure my project as below:

forum/ 
    core/
        api/ // API Defines core types and interfaces, or business logic if you will
            users.go
            posts.go 
            ...  // There maybe sub packages here too based on how complex it becomes, tough I have no need for it yet.
        db/ //  DB Implements interfaces in /api, they are dumb, mostly just CRUD's
            mysql/ 
    web/
        internal/ // Usually I keep the handlers inside internal directory 
            handlers/ // I don't have a strong opinion about this, I could keep this outside.
                users.go // But this let's users know it's only meant to be used inside web/
                posts.go
        files/ // the usual stuff
            static/ // public directory
                css/
                js/
            templates/
                index.html
                users/
                    list.html
        main.go // setup's routers, mount handlers to HTTP endpoints, storage repository interfaces from api/ to db/mysql/ or whatever and pass them to handlers on initialization.
    cmd/ // Other project related command line utilities go in here
        cli.go
    extended/ // I keep other unrelated packages that are unrelated to core functionalities and doesn't really belong to web application in an extended directory organized by their functionality, such as various ways of handling file uploads, S3, local file uploads  or session handling code with various back-end support.
        ...

This structure maybe overkill for simple applications, but I use it for small projects too. Makes it easy to reason about code. Go’s interface{} works great with this approach.

I’m still not sure what problems it will cause in the long run, so far it’s working great. I’d be happy if someone could give *pointers to improve this structure.