Pretty sure I wouldn’t call what I am doing for pocketgophers.com as “best”, but it seems to be working. Some of what I am doing “works” because, at this point, I only need a single process running on a single VPS. The VPS is running Ubuntu 16.04 LTS x86_64. When it starts making money I will build up a more failure resistant solution. I am also the only developer, so I don’t need to worry about access control for other devs.
If you see something I am doing wrong, please comment and help me make it right.
My main goal was to be able to git push any changes (code or content) to production and have it update and become live. The server process should also start when the VPS starts to recover from reboots (caused by updates, failures, etc.). Since I will be sharing the trickier setup details, I should note that this server server more than just pocketgophers.com, and as such is called sites.
The first component in this setup is github.com/jpillora/overseer, which handles graceful restarts (and in the future, self-upgrades) in a way that does not stop the process itself. This is done by using a small amount of code to oversee child processes that do the actual work. Most of the application code can be changed without needing to restart the overseer. Exceptions are things like which ports are listened to as the overseer needs to know that to hand them over from from child process to the next (upgraded) one. Overseer works nicely with supervisordbecause the overseer process is not restarting all the time and is also forwards stdin and stdout from the active child process to handle logs. The use of both github.com/jpillora/overseer and supervisord is pretty basic and the docs for each will lead the way.
The interaction between development and production is done with two, bare, git repositories on the VPS, which I will refer to as development.git and production.git. Only production.git needs to be on the VPS; development.git is currently there for convenience. Local development is started by cloning development.git and then adding production.git as a remote named production. I only push to production when I want to deploy, all other changes are kept in sync using development.git.
On the VPS, my application runs out of GOPATH, which I have set to $HOME. That version is cloned from development.git (also on the VPS). I use govendor to keep all my dependencies in git. This keeps my development and production builds consistent.
I use git’s post-update hook to respond to pushes to production.git My development.git/hooks/post-update is:
#!/bin/zsh
. $HOME/.zshrc
pushd $HOME/src/sites
export GIT_DIR=`pwd`/.git
echo "updating" &&
git pull --ff-only &&
./update
popd
This is meant to do as little as possible for the update process because it is the hardest to update. Its purpose it to update the application’s working copy with whatever was just pushed to production and then run the application’s update script. Since the pull happens before update is ran, the updated update script will run. The trickiest part was finding out that GIT_DIR needed to be reset to the application’s .git; while running as a hook it was set to development.git, which is usually what you want.
The update script:
#!/bin/sh
echo "building"&&
go install &&
echo "setting permissions"&&
sudo setcap cap_net_bind_service=+ep /home/alaster/bin/sites &&
echo "restarting"&&
kill -USR2 `sudo supervisorctl pid sites`
Notice that all the commands are chained together with &&. This makes it so that if any of them fail, the server won’t be restarted (as that is the last line). You should also know that the output from this script is sent to the git client doing the push, this lets me see what is happening and discover what went wrong.
There are two commands here using sudo. The first, setcap, gives the installed binary the necessary permissions to bind to ports 80 and 443 that normally requires superuser privileges. This must be done every time the binary is updated so that the next time the VPS reboots, the correct permissions are available to start the application. Updating the running application does not need this to be done because the overseer process holds the bindings for the updated child. Setting these permissions also allows the application to run as a user instead of root. The second gets the pid of the overseer process. Sending USR2 to the overseer tells it to start a new child process using the updated binary. Permission to run only these two commands without requiring a password (which would require user interaction) is given in /etc/sudoers:
alaster ALL = NOPASSWD: /sbin/setcap cap_net_bind_service=+ep /home/alaster/bin/sites
alaster ALL = NOPASSWD: /usr/bin/supervisorctl pid sites
I think that covers everything. Questions, comments, and improvements are welcome.