List and download dependencies for go generate before building?

[acmiel (Adam Cmiel)]

Hi! I have a potentially curious issue with pre-downloading dependencies for a network-isolated container build.

What I’m trying to do:

• pre-download dependencies via go mod download into a local directory
• mount the directory into a container build, set GOMODCACHE to point to this directory
• build the container without access to the internet

The problem: part of the build process is running go generate. This command wants to download additional modules which were not downloaded by go mod download. That fails because there’s no internet access.

go: downloading golang.org/x/tools v0.1.13-0.20220804200503-81c7dc4e4efa
go: downloading github.com/olekukonko/tablewriter v0.0.5
/tmp/gocache/entgo.io/ent@v0.11.3/cmd/internal/printer/printer.go:16:2: github.com/olekukonko/tablewriter@v0.0.5: Get "https://proxy.golang.org/github.com/olekukonko/tablewriter/@v/v0.0.5.zip": dial tcp: lookup proxy.golang.org: i/o timeout
/tmp/gocache/entgo.io/ent@v0.11.3/entc/load/load.go:27:2: golang.org/x/tools@v0.1.13-0.20220804200503-81c7dc4e4efa: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout
/tmp/gocache/entgo.io/ent@v0.11.3/entc/gen/graph.go:25:2: golang.org/x/tools@v0.1.13-0.20220804200503-81c7dc4e4efa: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout
storage/ent/generate.go:3: running "go": exit status 1

Reproducer (with go1.19.2):

Prepare the cache:

git clone https://github.com/dexidp/dex
cd dex
git checkout v2.35.1
env GOMODCACHE=$(realpath ../gocache) go mod download
cd ..

Containerfile:

FROM golang:1.19.1-alpine3.16 AS builder

# alpine-sdk would be needed if we could get past the download step
# RUN apk add alpine-sdk

COPY ./dex /src/dex
WORKDIR /src/dex

ARG GOMODCACHE=/tmp/gocache

RUN export GOMODCACHE="$GOMODCACHE" && \
    export GOOS=linux && \
    go generate -mod=mod /src/dex/storage/ent/

Build:

sudo podman network create --internal internal
sudo podman build --network=internal -v $(realpath ./gocache):/tmp/gocache:Z --build-arg GOMODCACHE=/tmp/gocache .

Possible solutions?

The ultimate goal is to make the container build as secure and reproducible as can be. In fact, this is part of a build system that does roughly as described: pre-downloads dependencies and then builds the container with network isolation.

With that in mind, I’ve found two sub-optimal solutions:

• prior to the build, run go generate in addition to go mod download
• use go mod download all instead (downloads the generate deps, not sure why)

Problem with both: the go.sum file for the example project does not include the additional dependencies. That means the extra downloads are not verified.

Problem with go generate: the host system doesn’t know what parameters to pass to go generate. It also may not have various non-golang dependencies needed to succesfully generate.

Problem with download all: downloads even more dependencies, most of which are not needed even for go generate.

Are there any other ways I missed to download the go generate dependencies? The ideal solution would be to have them included in the go.mod and go.sum of the project. However, this diverges from the usual golang dev workflow. Even running go mod tidy removes them again.