I am a newbie to Go. I clone a Go code base - they use vendor to store their dependencies. I added a custom package to the vendor folder, but when I do go build mod=vendor i get cannot find module providing package <package which is present in vendor dir> .
Also, It’s been terrible manually copying packages from $GOPATH/pkg/mod to this vendor directory. It’s never-ending. How can I convert this “vendor” mode to module mode and include the custom package I wrote earlier? How can I finally build this?
I know this is a basic question, but GoLang’s building/packaging has been so convoluted, that I am in the process of giving up! Please help!
Thanks!
Unfortunately, this is not a simple question but don’t worry about it.
If you’re using module, you should not use vendor anymore. It confuses everyone. Hence, the goal is to go build without the mod=vendor.
Depending on urgency, I propose 2 ways.
Short Term Mitigation
vendor has 1 main effect on overall repository that requires consideration: it influences the local import across the source codes. E.g.:
Instead of importing the package like import "github.com/XYZ/abc", placing that package in vendor directory vendor/abc allows the source codes to import package like import "abc".
Hence, we need to workaround it. Fortunately, go.mod has a replace clause for you to replace any import clause in your source codes to a local directory. Based on the example above,
replace (
abc => ./vendor/abc
...
)
Once you replace all the vendor dependencies accordingly, test run again without the mod=vendor argument. If everything works fine, you’re good to go for now.
Long Term Migration
For long term migrations, you need to change the source codes import clauses back to its original repository point. This will eliminate the dependency on vendor remote packages. As for non-remote vendor packages, you can create an independent module repository for it and import back to the project.
When I say long term, it means it takes time for the migration depending on the size of the repository and how widespread the vendor import statements in all the source codes. Consider it a refactor effort.
Here, couchbase/api/couchbase-cluster/v1 is the only custom, self written package - that itself might have some dependencies which are available on github. But when I run
bash-3.2$ GOOS=linux GOARCH=amd64 go build ./cmd/webhook-server/
This is not a valid module path. A valid one should be something like: github.com/ABC/XYZ/admission-controller-webhook-demo-master. This is essentially the import path for admission-controller-webhook-demo-master.
You need to go into ./vendor/couchbase/api/couchbase-cluster/v1 and initialize another go.mod like how you did for above. Yes, this is another module. The name should be: couchbase/api/couchbase-cluster/v1 until you push it to its dedicated repository.
The invalid module path is about the replace line - not the name of the module.
I don’t understand if I should push the custom module to a github repo? Also, should what should the import statement for the custom private module be in the Go code?
Please help! Can you write steps on how to get this done? even a rough outline would be so helpful!
The name is a problem since you’re essentially locking the module into your local machine. It works mainly due to the git VCS mechanism for now.
For long term, you should push that ./vendor/couchbase/api/couchbase-cluster into its own dedicated repo. It’s another module so it needs its own go.mod.
For now, let’s focus on short term.
Is the replace path correct? as in, is the package inside your {main repo}/vendor/couchbase/api/couchbase-cluster/v1 or {main repo}/vendor/couchbase/api/couchbase-cluster?
The compliant is about the replace is pointing at an invalid relative path.
The migration checklist is quite straightforward:
freeze the version using your VCS (git commit).
initialize a go.mod.
review all vendor external dependencies and add them into require clause.
replace all vendor internal dependencies and add them into replace clause.
build and test the result. It must reproduce step 1 results.
freeze the version using your VCS (git commit).
migrate all internal dependencies into its dedicated modules externally.
correct all the import statements to use the external modules created in step 7.
build and test the result. It must reproduce step 1 results.
freeze the version using your VCS (git commit).
remove the replace clause.
build and test the result. It must reproduce step 1 results.
delete the vendor directory.
freeze the version using your VCS (git commit).
Treat yourself with a double cheese burger + cola [ ENDED ].
Beyond step 6 are long term implementations. You’re in Step 4. If you want to jump straight to long term, jump to Step 7.
I am in $GOPATH/src/admission-controller-webhook-demo-master (this is my project)
I do export GO111MODULE=on
I run go mod init admission-controller-webhook-demo-master
bash-3.2$ go mod init admission-controller-webhook-demo-master
go: creating new go.mod: module admission-controller-webhook-demo-master
go: copying requirements from Gopkg.lock
Now, when I cat go.mod here, I get the below
bash-3.2$ cat go.mod
module admission-controller-webhook-demo-master
go 1.12
require (some gihub and k8s stuff I cannot put here because I am new user :frowning: )
I cd to my wonderful private, self written, non-public package. cd vendor/couchbase/api/couchbase-cluster/v1/
List directories
bash-3.2$ ls -lrt
total 40
-rw-r–r-- 1 sanupin 1368638373 610 Oct 7 15:01 register.go
-rw-r–r-- 1 sanupin 1368638373 154 Oct 7 15:01 doc.go
-rw-r–r-- 1 sanupin 1368638373 3327 Oct 7 15:01 zz_generated.openapi.go
-rw-r–r-- 1 sanupin 1368638373 1940 Oct 7 15:12 couchbasecluster_types.go
-rw-r–r-- 1 sanupin 1368638373 3046 Oct 7 15:12 zz_generated.deepcopy.go
initialize the module here
bash-3.2$ go mod init `pwd`
go: creating new go.mod: module /Users/sanupin/go-workspace/src/admission-controller-webhook-demo-master/vendor/couchbase/api/couchbase-cluster/v1
I edit the module name in go.mod here to be “couchbase/api/couchbase-cluster/v1”
module couchbase/api/couchbase-cluster/v1
go 1.12
I go back to $GOPATH/src/admission-controller-webhook-demo-master
I edit the go.mod to include the “replace” directive - now go.mod looks like below. I only added the replace directive.
module admission-controller-webhook-demo-master
go 1.12
require (some gihub and k8s stuff I cannot put here because I am new user :frowning: )
replace github.com/ABC/XYZ/couchbasecluster => ./vendor/couchbase/api/couchbase-cluster/v1
In my actual golang code (sometimes, i forget it ), when I have to import my wonderful private, self written, non-public package., I import it as below (check bold line)
import (
"errors"
"fmt"
<some k8s pkg>
corev1 "<i can only put 2 links in a post>"
metav1 "<i can only put 2 links in a post>"
**couchbasev1 "github.com/ABC/XYZ/couchbasecluster"**
"log"
"net/http"
"path/filepath"
)
Now I build SUCCESSFULLY bash-3.2$ GOOS=linux GOARCH=amd64 go build ./cmd/webhook-server/
Holding off on the cheese burger and celebrations partly because I am vegetarian, but also partly because I wanted to confirm with you if this was the short term solution you had in mind…?
), when I have to import my wonderful private, self written, non-public package., I import it as below (check bold line)
replace all