Versioning Libraries - Vanity URLs? Tags?

Vendoring and tagging typically address the consumption of libraries from an application perspective, but what about versioning libraries across breaking changes?

As a library author I can SemVer all I like, but since go get and related tools operate on HEAD, most casual (non-org) library users are still going to suffer breakage when I make a breaking change.

So, in combination with SemVer (via Git tags), what are the approaches for maintaining two (or more) versions of a library?

  1. Versioned import URLs similar to gopkg.in - e.g. example.com/v1/pkgname vs. example.com/v2/pkgname via an external service until issue 10913 is resolved to allow the go-import <meta> tag to specify branches/revisions.
  2. Separate repositories with separate import paths e.g. github.com/elithrar/pkg and github.com/elithrar/ctx-pkg? (which doesn’t have to be distinct from option 1 if you use the meta tag to point to these)
  3. Something else entirely?

For context, I’m attempting to maintain the “v1” version of a few HTTP libraries using gorilla/context whilst also providing a “v2” that uses net/context instead. Making it easy for new Gophers—who aren’t going to git subtree a specific tag or branch into their /vendor/ dir—to use the package they need without too much confusion is a top priority.

Right now I’m leaning towards vanitydomain.com/v{N}/pkgname pointing to different GitHub repositories, which works with both the go tool and GoDoc from today.

Open to suggestions & feedback here.

1 Like

I believe that incompatible versions of the same library should use different import paths (unless your library is experimental/under active development or you can force all your users to update their code).

And this is not only dictated by how go get works. Consider the following situations:

  1. A company has been using your library to develop a number of applications. They also have a policy to vendor all 3-rd party dependencies (e.g. https://blog.gopheracademy.com/advent-2015/go-in-a-monorepo/). When you release a new backward-incompatible version of your library they want to start using it for the new apps. They do not want to update existing apps (at least now). By keeping the same import path you force your users to change it for you.

  2. My app uses libraries A and B. Both of these libraries depend on incompatible versions of library C. If these versions of library C use the same import path I will have hard time rewriting paths, vendoring custom copies of different libraries, etc.

Note that problem 2 is not specific to Go. It exists in Java, Haskell, Rust, etc.

1 Like

I definitely agree with the need to have separate URLs for different versions, especially as there isn’t a standard versioning scheme or manifest (i.e. github.com/user/pkg@v3).

Although I’d still like to have a single underlying repository with branches exposed by different import URLs (as vanity domains), the most practical approach today seems to be to just version the URL ala the mgo driver and others: project.ext/v1/pkgname.

You can at least do this via GitHub Pages and Jekyll, which makes it cheap, fast and arguably more reliable than the GitHub repo itself.

1 Like

If you give multiple versions of your library different names, the multiple versions of those libraries will be present in people’s applications. Have a look at the various vendor/ discussions to see why this is a flawed approach.

http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#1

TL,DR: things which are the same (the same library) must have the same name.

Thanks for the reply Dave. I read your presentation back when you originally linked to it, and as a user of gb for my projects, I agree with the approach for those.

But gb (nor any other tool) doesn’t solve the problem for library authors: how do I intentionally break my library’s API? (assume I correctly SemVer my releases)

  1. Just break it and tell users of the library to vendor the older version it if they cared about what they consume? (a hard truth, but not pragmatic)
  2. Tell newcomers to the language to install gb|godep|glide|fotm and ignore go get despite all of the documentation that exists around it?
  3. ???

Even vendoring now isn’t an A++ experience—copying deps into /vendor without tracking the revision or using git subtree (or similar) is a pretty average experience and makes it hard to selectively update your dependencies (“new bugfix release? What rev do I have now?”). There may also be reasons (as an author) to update an older branch (bugfixes, secfixes) whilst having master reflect the ‘latest’ package API.

I do agree that putting the version in the import path has problems of its own too—although you do get a compile time error if you import both packages in the same file, you’re not prevented from importing both versions in different files of the same package. This could cause unintended behaviour if (e.g.) goimports automatically adds the wrong package for you and the API break was subtle.

Every option thus far is imperfect (but what in the world of PL dep mgmt isn’t?), which makes me feel that versions-in-the-import-path are just another option you need to consider relative to your library consumers (newbies? advanced users? in-between?).

(I’d feel pretty irresponsible if I broke (e.g.) gorilla/csrf because I’m fairly confident that the majority of its users are not vendoring it)

Library writers must not vendor dependencies. We’ve discovered that with tools like glide and govendor that to make the vendor experiment work reliably, the consumer of a library must flatten the imported dependencies into its project, and that process continues recursively.

I don’t think using versioned import paths is a solution, even a stopgap one, because they have the same problem as vendored packages – the same package is known by different names.

Here is an example of what can go wrong, https://github.com/mattfarina/golang-broken-vendor.

1 Like

Just to be clear: I am not advocating for library authors to vendor their own deps.

I’m saying that existing tools do nothing to help library authors assist downstream users consuming their library—I can git tag v1.0/v2.0/v3.0 to my hearts content, but consumers have to know to manually pin it or hope their competing tool does it for them. go get users are still left out in the lurch, as is the ability to use godoc.org. That’s last one isn’t a Go toolset problem, but it says a lot about how we treat multiple-versions-per-package today.

If versioned import paths are not a solution (ever) are you suggesting I just break the API (substantially), tag the new version and hope a blurb in a README/GH issue is enough?

At the moment as a package author, you have no good options; sorry. As a package author you have no programatic way of communicating with your users; you can only do so via release notes. I wish I had better news.

Even using something like a gopkg.in is insufficient because that only guarentees your downstream users are consuming something that looks like .v7 of your package, you cannot communicate at any higher granularity. So unless you want to adopt the chrome/mozilla model of only using major release numbers, that method has very little utility.

This is the worst problem in the Go ecosystem at the moment, and one that has probably already done long term damage to the adoption of the language.

At the moment, my advice is to encourage your downstream users to vendor the specific version of a package they are using. I’d also recommend they use something like gb if they are developing a project (by gb’s definition of a project).

1 Like

This is going to be part opinion, part practical, part history, and part community.

I’m discovering that people are quite unhappy with the state of package dependency management in Go. I’ve observed a couple reasons for this and they highlight how we got to where we are (which I consider to be broken but now on that in a moment).

  • Go was designed by C/C++ developers. They’ve worked on (and designed) other languages over the years and are quite smart at language design. But, if you look at the supporting tooling, especially around dependencies, those languages are not ideal. Many Go developers are coming from JavaScript, Java, PHP, Python, and numerous other languages that have solved the dependency management problem. You’ve got people coming from differing expectations.
  • Go was designed at Google where they write quite a bit of proprietary code and work in a monorepo. This is different from the open things we talk about being open source and sharing lots of smaller projects with each other. Go prioritized the Google style. Many of us approaching dependencies are trying to prioritize the other.

Note, I’ve spoken with (ex)googlers who have described dependency rot issues in their codebases because of poor dependency handling. I wouldn’t call their setup something to idolize.

The current implementation by the go tool is broken for a few reasons.

  1. There is no version locking. Two people use go get to retrieve dependencies at different times and they can get different versions. This leads to a lack of ability to create reproducible builds.
  2. When you have a dependency tree with different version needs there is no way to resolve that and easily handle updates. This is currently a manual process. Every library can’t store their own dependencies in a vendor/ folder because it can cause issues. There is no “official” way to handle this.
  3. The discussion is around using different paths for different versions of a package. This design is for monorepos. In the distributed open source world it doesn’t work well. Can you imagine every version of a package needing a new GitHub repo? That’s not going to happen. No one does this. Using a different repo for different versions would be to ask developers to consider Go a unicorn to fit the monorepo style they use at Google.

The solution every other modern language has gone to is SemVer plus a package manager.

Because the Go leadership stems from Google and they don’t live the problems we need to solve I think this is up to the community to solve.

I’ve been working, like so many others, on this problem. Glide is a package manager like Cargo (Rust), Composer (PHP), Pip (Python), Maven (Java), npm (JavaScript), etc.

It does a few things:

  • SemVer and Lockfiles - deals with version numbers, ranges, and more
  • Works with aliases and private repos.
  • Can pull information from Godep, GB, and GPM in addition to glide.yaml files.
  • Is in line with the other package managers listed above.

The default Go way is insufficient and we can do better. Or, that’s my 2 cents.

2 Likes

The dependency handling and confusing nature of the GOPATH are both causing problems. There is a lot of potential for another toolchain around the language or another language that tackles these problems well. People are definitely unhappy.

Even with that, if the Go team at Google stays out of the way there is an opportunity to undo some of the damage. Guess I’m still a little optimistic.

1 Like

You’re right there. gopkg.in also suffers the same problem as versioning your own import paths: I can (mistakenly) import gopkg.in/natefinch/lumberjack.v1 and gopkg.in/natefinch/lumberjack.v2 into the same library—gopkg.in just outsources/automates the infrastructure.

@mattfarina — thanks for the comment. I’ve been following the vendoring/packaging discussion for a long while (I picked up Go around 1.1?), and am aware of the multitude of competing tools, manifest formats, etc. that exist to tackle the problem in different ways. From a package user POV you could argue that the choice is good, but from an library author POV you’re gonna have a bad time.

I think the only workable solution I have (for my particular set of reqs) thus far is to just version the URLs, take care to avoid the traps surrounding it, and document the failure cases thoroughly until a more pragmatic solution rears its head.

Which—as you both point out—is a real shame, because I shouldn’t have had to ask this in the first place!

Here are a few things that I’m doing to try to improve the situation.

  1. Obviously gb, but I realised that isnt for everyone.
  2. Proposal 12302 died in committee, but I intend to implement it in gb, with a new plugin that downloads released packages on demand.

There is still the extrmemly contentious problem of how can one package declare a dependency on a version of another. This is a very devisive issue and one the Go team have shown no leadership on, so I’m sticking to the problems that can be addressed by the gb project approach.

3 Likes

I would ask that you also tag using SemVer.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.