I maintain miglite, a small migration tool for Go projects that prefer plain SQL files. The v0.8.0 release adds fs.FS support, mainly for services that want to embed migrations with go:embed:
//go:embed migrations/*.sql
var migrationFS embed.FS
mig.SetFS(migrationFS).SetSqlDB(db)
The migration path is an io/fs logical path, and SetFS(nil) switches back to the local filesystem. The CLI commands and file format stay unchanged.
The same update fixes three behaviours that were easy to miss: --skip-err now continues but returns a failure, injected *sql.DB values are not closed by miglite, and a file without DOWN is reported as skipped.
For small services, how do you decide between embedding migrations and shipping a separate directory? I would especially appreciate feedback on the fs.FS API.
I would never use embedding for plain sql files, because the chew memory without the option toclean that.
Normally for each file you read, you can clean up afterwards.
The disadvantage is, that you have to ship the separate files with the executable.
(but perhaps you can make them available via some cloud repo?)
When deploying to standard container-y cloud solutions like Google Cloud Run, I generally prefer to ship my migrations embedded in my binaries. The less schlepping files around in docker via COPY I can do, the better.
Re: memory pressure, the ramifications in most of my apps is almost nil. My understanding is that it increases virtual memory because of increased binary size (by exactly the increase in binary size), but the resident memory is mostly unaffected. Important caveat: when talking migrations, I’m not talking about seed data. I just scanned a few of my project folders and most of the schema ranges from 18kb (small project, 30-ish migrations) to 300kb (medium project with 100-ish tables). I imagine on some of my larger projects the schema might hit 1mb but that is a lot of schema/tables and not a lot of virtual memory.
For inserting seed/dev data (where file size is going to be larger) I usually have some other mechanism by which I do that. Usually I just build a binary in my /cmd folder that either uses on-disk scripts or scripts stored in object storage (AKA cheap cloud storage). That is usually a once-per-environment type thing for local dev setup, etc. For test/staging environments I usually restore from last night’s prod backup.
All of this is to say, yes, I personally think embedding migrations is great. It keeps things self-contained. You don’t have to wrestle with a real filesystem and permissions related to that (which matters to me in some sandboxed environments). Etc.
My only question is: why build your own when it seems like goose has become the de facto standard for this type of migrations library/tooling?