Hi! I have a potentially curious issue with pre-downloading dependencies for a network-isolated container build.
What I’m trying to do:
- pre-download dependencies via
go mod downloadinto a local directory
- mount the directory into a container build, set GOMODCACHE to point to this directory
- build the container without access to the internet
The problem: part of the build process is running
go generate. This command wants to download additional modules which were not downloaded by go mod download. That fails because there’s no internet access.
go: downloading golang.org/x/tools v0.1.13-0.20220804200503-81c7dc4e4efa go: downloading github.com/olekukonko/tablewriter v0.0.5 /firstname.lastname@example.org/cmd/internal/printer/printer.go:16:2: email@example.com: Get "https://proxy.golang.org/github.com/olekukonko/tablewriter/@v/v0.0.5.zip": dial tcp: lookup proxy.golang.org: i/o timeout /firstname.lastname@example.org/entc/load/load.go:27:2: email@example.com: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout /firstname.lastname@example.org/entc/gen/graph.go:25:2: email@example.com: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout storage/ent/generate.go:3: running "go": exit status 1
Reproducer (with go1.19.2):
Prepare the cache:
git clone https://github.com/dexidp/dex cd dex git checkout v2.35.1 env GOMODCACHE=$(realpath ../gocache) go mod download cd ..
FROM golang:1.19.1-alpine3.16 AS builder # alpine-sdk would be needed if we could get past the download step # RUN apk add alpine-sdk COPY ./dex /src/dex WORKDIR /src/dex ARG GOMODCACHE=/tmp/gocache RUN export GOMODCACHE="$GOMODCACHE" && \ export GOOS=linux && \ go generate -mod=mod /src/dex/storage/ent/
sudo podman network create --internal internal sudo podman build --network=internal -v $(realpath ./gocache):/tmp/gocache:Z --build-arg GOMODCACHE=/tmp/gocache .
The ultimate goal is to make the container build as secure and reproducible as can be. In fact, this is part of a build system that does roughly as described: pre-downloads dependencies and then builds the container with network isolation.
With that in mind, I’ve found two sub-optimal solutions:
- prior to the build, run
go generatein addition to
go mod download
go mod download allinstead (downloads the generate deps, not sure why)
Problem with both: the go.sum file for the example project does not include the additional dependencies. That means the extra downloads are not verified.
Problem with go generate: the host system doesn’t know what parameters to pass to go generate. It also may not have various non-golang dependencies needed to succesfully generate.
Problem with download all: downloads even more dependencies, most of which are not needed even for go generate.
Are there any other ways I missed to download the
go generate dependencies? The ideal solution would be to have them included in the
go.sum of the project. However, this diverges from the usual golang dev workflow. Even running
go mod tidy removes them again.