Go get stopped respecting ssh git.url.insteadOf?

I have a global git setting

[url "git@bitbucket.org:"]
    insteadOf = https://bitbucket.org/

and no local shadows or overrides. Some time recently, for myself, some others in my org, and all our automation pipelines, go get appears to have stopped respecting this setting. If I run go get -x bitbucket.org/organizationname/foo, what I see is:

# get https://api.bitbucket.org/2.0/repositories/organizationname/foo?fields=scm
# get https://api.bitbucket.org/2.0/repositories/organizationname/foo?fields=scm: 404 Not Found (0.568s)
# get https://proxy.golang.org/bitbucket.org/@v/list
# get https://proxy.golang.org/bitbucket.org/@v/list: 410 Gone (0.076s)
# get https://api.bitbucket.org/2.0/repositories/organizationname/foo?fields=scm
# get https://api.bitbucket.org/2.0/repositories/organizationname/foo?fields=scm: 404 Not Found (0.080s)
go get bitbucket.org/organizationname/foo: reading https://api.bitbucket.org/2.0/repositories/organizationname/foo?fields=scm: 404 Not Found

Go version is go1.16 darwin/amd64 (EDIT: specifically go1.16.0, which turned out to matter), and hasn’t changed recently AFAICT. I have confirmed that using git directly, e.g. git ls-remote git@bitbucket.org:organizationname/foo, works correctly and shows me the branches as expected. What is going on here?

Hi @jkop,

Two things pop up at a first glance.

  • The go get cmdline has no version tag. Do you run this command inside a main module where a go.mod exists? If you run the command outside a module, try appending @<version>, e.g. @v1.2.3 or @latest, to the module path.
  • The go get command also fails at the Go proxy. Is the module private? If so, check the GOPRIVATE setting. If the module is public, the 410 Gone status indicates that the module has not been cached by the proxy, at least not at the version the go get command requests. Again, adding a version suffix to the module path might help.
    In case the module is public (and thus should already be cached by the Go proxy), the problem is definitely not about ssh, because the Go proxy should happily deliver the module over HTTPS.

To test if HTTPS vs SSH is a problem here, you might want to disable the git setting and verify whether git succeeds fetching the repo over HTTPS.

That’s about all the ideas I have right now, I hope one of them is at least leading into the right direction.

I had also been using go1.16 darwin/amd64. I just started seeing the same error for updating a private module as posted above. My GOPRIVATE is set properly and using go get -x bitbucket.org/org/repo worked in the past.

I updated to go1.18 and it solved the issue, and go get works as expected for a private repo now.

So I’m not sure why that started being an issue for me, but upgrading seemed to fix it.

I just started seeing the same error for updating a private module as posted above. My GOPRIVATE is set properly and using go get -x bitbucket.org/org/repo worked in the past.

I updated to go1.18 and it solved the issue, and go get works as expected for a private repo now.

+1

Upgrading worked because Bitbucket, on only a few days of notice, stopped returning 403s for private repositories which were not properly authenticated, replacing them with 404s. Their reasoning, such as it is, is described at https://community.atlassian.com/t5/Bitbucket-articles/Changes-to-Bitbucket-API-Requires-Latest-Version-of-Go/ba-p/1975819. No explanation is given for why they felt it was reasonable to announce a breaking change on Wednesday on a blog no one reads, and adopt it the following Thursday without ability to opt out or useful error messages.

Go versions 1.16.14, 1.17.7, and 1.18.x updated the behavior of go get so that it responds the same way to 404s as it does to 403s. This fixes the problem. Good luck to anyone who suddenly needs to update their Go version to keep the lights on, and cannot run any testing automation while they do it.

Update: Bitbucket has realized this has made a lot of people very angry and been widely regarded as a bad move(x), and rolled back the change for the time being. Hopefully they will do the standard steps of deprecation and issuing warnings before they implement it again. Still, updating to 1.16.14/1.17.7/1.18.x in the meantime is probably wise.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.