`go mod download && go mod vendor && go mod tidy` often requires 2x runs to succeed

Technical Discussion
1

Go mod often fails the first time. It generates spurious errors like missing go.sum entry. And so I often end up running this sequence twice in a row:

  1. go mod download
  2. go mod vendor (recommended)
  3. go mod tidy

Can Go please fix the mod system? Not only is it tiring to repeat the sequence twice, doubling my labor across very many Go projects. But it’s also silly to have to invoke three distinct commands to update the Go caching system.

There should really be a simple command like go mod refresh that performs each of these low level steps. It should read an optional boolean parameter that controls whether or not to vendor, from the go.mod configuration file. It should never requiring repeating the command again to resolve low level recoverable errors.

Yet another case in favor of a single, unified refresh command is the fact that multiple commands introduce subtle bugs into larger scripts. POSIX shells may apply && and/or set -e to ensure that genuine errors propagate through downstream systems, such as CI/CD jobs. But this syntax is not portable beyond *nix environments. Command Prompt and PowerShell struggle to reliably bubble up accurate exit codes. Combining these operations into a single command makes scripts that run go mod… commands much more reliable. Without having to resort to shoving everything into a more verbose mage Go task.

2

Hello there. You have this error, because the sequence of steps is wrong. Just add module you need into go.mod (or as import into the file you’re working on) and simply run go mod tidy. It will download everything on its own and add entry into go.sum. After you can run go mod vendor if you need this.

1 Like
3

Agreed. I have never seen or had an issue with just using go mod tidy.

4

simply run go mod tidy . It will download everything on its own and add entry into go.sum .

On my machine, go mod vendor often complains that I need to manually go mod download beforehand.

5

go mod tidy should suffice. But you have to download the modules to be able to vendor them. Also it checks vendor/modules.txt for discrepancies with your go.mod and will report errors:

go mod vendor also creates the file vendor/modules.txt that contains a list of vendored packages and the module versions they were copied from. When vendoring is enabled, this manifest is used as a source of module version information, as reported by go list -m and go version -m. When the go command reads vendor/modules.txt, it checks that the module versions are consistent with go.mod. If go.mod has changed since vendor/modules.txt was generated, the go command will report an error. go mod vendor should be run again to update the vendor directory.

But still, you should be good running go mod tidy once (clean up/download dependencies) then go mod vendor (vendor the dependencies we just downloaded).

6

go mod tidy followed by go mod vendor appears to work.

However, again, having to run two separate commands to update the dependency cache is risky, especially in terms of OS/shell portability. The programming language stands to improve by adopting a boolean option in go.mod that can automatically trigger go mod vendor each time go mod tidy runs.

Unknown if there are any other scenarios where a separate go mod download may be required.

7

Vendor is not a recommended behavior. It is required if you want to store specific versions of packages or add changes into them for your needs. It is very situational and every developer would prefer to have an opportunity to control this. And again it looks like you don’t understand where cache is located. Cache is a global storage which you update with go mod tidy. go mod vendor just updates local project’s vendor folder with updates you’ve downloaded into cache.

8

How is this risky?

You could always create a proposal. I don’t know how much traction it would get, but, it would be a relatively small change I would imagine.

9

How is it risky?

Fair question.

In POSIX shells such as bash, modern sh, ksh, and zsh, you can use double ampersand and/or set -e to ensure that no exit code errors from commands are dropped. But most people will use the unsafe forms: semicolon or newline. As a result, any scripts or interactive sessions are likely to ignore errors and run successive commands.

That’s without getting into pipefail, IFS, undefined variables, globs, and yet other shell pitfalls. Or trap semantics. Anything running in a CLI context is inherently a fragile piece of ■■■■, so simplicity in the command characters is a blessing. Not simplicity in app behavior, but rather simplicity in which conjunctions or other nonsense are involved.

The error remains hidden. The already corrupt state may become even more corrupt. Wolves bay. The earth trembles.

Native Windows shells do not consistently support the POSIX sh conjunction nor safety flags. Attempts to apply them tend to break Command Prompt / MS-DOS bat(ch) or PowerShell commands.

Single quotes, escape syntax, POSIX vs GNU vs BSD find, the names of the stdio handles… such details result in thousands of “helpful” Stack Overflow posts of command snippets breaking for all practical purposes. They’re vendor locked and fail to consider a significant portion of the state space.

Fish, (t)csh, ion, and other non-POSIX shells likewise do not support 100% of these safety options.

Two-step command patterns tend to break safety, portability, or both.

As a workaround, you can shove them in a POSIX makefile. Or a mage file. (Mage lets you write tasks in pure, normally portable Go code.) However, even make/mage contribute to a larger attack surface than strictly necessary, complicate the build process, and bog down CI/CD jobs.

A unified command that performs more operations, helps to encourage safe, portable build steps.

10

I feel like you’re vastly overstating how “risky” this is. Is it risky to git add files to stage them then git commit them?

Most of the time, a developer will be running these commands to get dependencies and vendor them. Since Go projects have a tendency to use fewer dependencies than most, we aren’t running go mod tidy very often in my experience. But if something breaks, we will of course need to fix it before we go mod vendor. I don’t see how having one command that does both is of any benefit. And if you wanted a script to run both:

#!/bin/bash

# Exit immediately on failure
set -e

# Tidy our go mod and download dependencies
go mod tidy
# Vendor our dependencies
go mod vendor

By the time your CI/CD job is running, your stuff should be vendored and vetted by whatever developer updated your dependencies. And again - if you were going to run multiple commands in a CI/CD job runner, every one of them that I’m aware of fail immediately if a command fails. For example in a gitlab job runner:

build:api:
  image: golang:1.22
  stage: build
  script:
    - go mod tidy # Would fail job immediately on error
    - go mod vendor # Why would a CI/CD pipeline do this?

But even then - you wouldn’t have an attack surface. You would have a broken build.

If you’re talking about updating dependencies to newer versions with security fixes, usually this is not handled by CI/CD pipelines (nor would you want it to be!). For example, GitHub has dependabot which scans your dependencies and creates automated pull requests to bump them to later versions. But the pull requests are reviewed by a human developer.

1 Like
11

Yes.

If only -a supported new files.