Converting vendor mode to module


(Satish Anupindi) #1

Hi,

I am a newbie to Go. I clone a Go code base - they use vendor to store their dependencies. I added a custom package to the vendor folder, but when I do go build mod=vendor i get cannot find module providing package <package which is present in vendor dir> .

Also, It’s been terrible manually copying packages from $GOPATH/pkg/mod to this vendor directory. It’s never-ending. How can I convert this “vendor” mode to module mode and include the custom package I wrote earlier? How can I finally build this?

I know this is a basic question, but GoLang’s building/packaging has been so convoluted, that I am in the process of giving up! Please help!
Thanks!


(Holloway) #2

Welcome to Golang Bridge! :rocket::fireworks:

Unfortunately, this is not a simple question but don’t worry about it.

If you’re using module, you should not use vendor anymore. It confuses everyone. Hence, the goal is to go build without the mod=vendor.

Depending on urgency, I propose 2 ways.


Short Term Mitigation

vendor has 1 main effect on overall repository that requires consideration: it influences the local import across the source codes. E.g.:

  1. Instead of importing the package like import "github.com/XYZ/abc", placing that package in vendor directory vendor/abc allows the source codes to import package like import "abc".

Hence, we need to workaround it. Fortunately, go.mod has a replace clause for you to replace any import clause in your source codes to a local directory. Based on the example above,

replace (
	abc => ./vendor/abc
	...
)

Once you replace all the vendor dependencies accordingly, test run again without the mod=vendor argument. If everything works fine, you’re good to go for now.


Long Term Migration

For long term migrations, you need to change the source codes import clauses back to its original repository point. This will eliminate the dependency on vendor remote packages. As for non-remote vendor packages, you can create an independent module repository for it and import back to the project.

When I say long term, it means it takes time for the migration depending on the size of the repository and how widespread the vendor import statements in all the source codes. Consider it a refactor effort.


(Satish Anupindi) #3

Hi,

If my go.mod looks like this :

module /Users/sanupin/Downloads/admission-controller-webhook-demo-master
  
go 1.12

require (
        github.com/ghodss/yaml v1.0.0
        github.com/gogo/protobuf v0.0.0-20171007142547-342cbe0a0415
        github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
        github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf
        github.com/json-iterator/go v1.1.5
        github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
        github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742
        github.com/spf13/pflag v1.0.3
        golang.org/x/net v0.0.0-20180124060956-0ed95abb35c4
        golang.org/x/text v0.3.0
        gopkg.in/inf.v0 v0.9.0
        gopkg.in/yaml.v2 v2.2.2
        k8s.io/api v0.0.0-20180308224125-73d903622b73
        k8s.io/apimachinery v0.0.0-20180228050457-302974c03f7e
)

replace (
        couchbase/api/couchbase-cluster/v1 => ./vendor/couchbase/api/couchbase-cluster/v1
)

Here, couchbase/api/couchbase-cluster/v1 is the only custom, self written package - that itself might have some dependencies which are available on github. But when I run

bash-3.2$ GOOS=linux GOARCH=amd64 go build ./cmd/webhook-server/

go: errors parsing go.mod:

/Users/sanupin/Downloads/admission-controller-webhook-demo-master/go.mod:23: invalid module path

What is wrong about my module path for it to complain?


(Holloway) #4

This is not a valid module path. A valid one should be something like: github.com/ABC/XYZ/admission-controller-webhook-demo-master. This is essentially the import path for admission-controller-webhook-demo-master.

You need to go into ./vendor/couchbase/api/couchbase-cluster/v1 and initialize another go.mod like how you did for above. Yes, this is another module. The name should be: couchbase/api/couchbase-cluster/v1 until you push it to its dedicated repository.


(Satish Anupindi) #5

Thanks Halloway, but I don’t understand.

The invalid module path is about the replace line - not the name of the module. :confused:

I don’t understand if I should push the custom module to a github repo? Also, should what should the import statement for the custom private module be in the Go code?

Please help! Can you write steps on how to get this done? even a rough outline would be so helpful!

Thank You!!


(Holloway) #6

The name is a problem since you’re essentially locking the module into your local machine. It works mainly due to the git VCS mechanism for now.

For long term, you should push that ./vendor/couchbase/api/couchbase-cluster into its own dedicated repo. It’s another module so it needs its own go.mod.

For now, let’s focus on short term.

Is the replace path correct? as in, is the package inside your {main repo}/vendor/couchbase/api/couchbase-cluster/v1 or {main repo}/vendor/couchbase/api/couchbase-cluster?

The compliant is about the replace is pointing at an invalid relative path.

The migration checklist is quite straightforward:

  1. [X] freeze the version using your VCS (git commit).
  2. [X] initialize a go.mod.
  3. [X] review all vendor external dependencies and add them into require clause.
  4. [ ] replace all vendor internal dependencies and add them into replace clause.
  5. [ ] build and test the result. It must reproduce step 1 results.
  6. [ ] freeze the version using your VCS (git commit).
  7. [ ] migrate all internal dependencies into its dedicated modules externally.
  8. [ ] correct all the import statements to use the external modules created in step 7.
  9. [ ] build and test the result. It must reproduce step 1 results.
  10. [ ] freeze the version using your VCS (git commit).
  11. [ ] remove the replace clause.
  12. [ ] build and test the result. It must reproduce step 1 results.
  13. [ ] delete the vendor directory.
  14. [ ] freeze the version using your VCS (git commit).
  15. [ ] Treat yourself with a double cheese burger + cola [ ENDED ].

Beyond step 6 are long term implementations. You’re in Step 4. If you want to jump straight to long term, jump to Step 7.


(Satish Anupindi) #7

Hi HollowayKeanho,

OK,

  1. I am in $GOPATH/src/admission-controller-webhook-demo-master (this is my project)
  2. I do export GO111MODULE=on
  3. I run go mod init admission-controller-webhook-demo-master

    bash-3.2$ go mod init admission-controller-webhook-demo-master

    go: creating new go.mod: module admission-controller-webhook-demo-master

    go: copying requirements from Gopkg.lock

Now, when I cat go.mod here, I get the below

bash-3.2$ cat go.mod 

module admission-controller-webhook-demo-master

go 1.12

require (some gihub and k8s stuff I cannot put here because I am new user :frowning: )
  1. I cd to my wonderful private, self written, non-public package.
    cd vendor/couchbase/api/couchbase-cluster/v1/

  2. List directories

bash-3.2$ ls -lrt
total 40
-rw-r–r-- 1 sanupin 1368638373 610 Oct 7 15:01 register.go
-rw-r–r-- 1 sanupin 1368638373 154 Oct 7 15:01 doc.go
-rw-r–r-- 1 sanupin 1368638373 3327 Oct 7 15:01 zz_generated.openapi.go
-rw-r–r-- 1 sanupin 1368638373 1940 Oct 7 15:12 couchbasecluster_types.go
-rw-r–r-- 1 sanupin 1368638373 3046 Oct 7 15:12 zz_generated.deepcopy.go

  1. initialize the module here

    bash-3.2$ go mod init `pwd`

    go: creating new go.mod: module /Users/sanupin/go-workspace/src/admission-controller-webhook-demo-master/vendor/couchbase/api/couchbase-cluster/v1

  2. I edit the module name in go.mod here to be “couchbase/api/couchbase-cluster/v1”

module couchbase/api/couchbase-cluster/v1

go 1.12

  1. I go back to $GOPATH/src/admission-controller-webhook-demo-master

  2. I edit the go.mod to include the “replace” directive - now go.mod looks like below. I only added the replace directive.

    module admission-controller-webhook-demo-master
    
    go 1.12
    
    require (some gihub and k8s stuff I cannot put here because I am new user :frowning: )
    
    replace github.com/ABC/XYZ/couchbasecluster => ./vendor/couchbase/api/couchbase-cluster/v1
    
  3. In my actual golang code (sometimes, i forget it :slight_smile: ), when I have to import my wonderful private, self written, non-public package., I import it as below (check bold line)

    import (
    
    "errors"
    
    "fmt"
    
    <some k8s pkg>
    
    corev1 "<i can only put 2 links in a post>"
    
    metav1 "<i can only put 2 links in a post>"
    
    **couchbasev1 "github.com/ABC/XYZ/couchbasecluster"**
    
    "log"
    
    "net/http"
    
    "path/filepath"
    
    ) 
    
  4. Now I build SUCCESSFULLY :tada::confetti_ball::trophy:
    bash-3.2$ GOOS=linux GOARCH=amd64 go build ./cmd/webhook-server/

  5. Holding off on the cheese burger and celebrations partly because I am vegetarian, but also partly because I wanted to confirm with you if this was the short term solution you had in mind…? :slight_smile:

Thank you, dear GoLang Angel!


(Holloway) #8

Très Bien! :+1:

Excellent! Job well done. :+1:

Whatever you did were item 4 and item 5. I suggest you do item 6 before partying. Short term plan stops at item 6 to be safe.


Your updated checklist now is:

  1. [X] freeze the version using your VCS (git commit).
  2. [X] initialize a go.mod .
  3. [X] review all vendor external dependencies and add them into require clause.
  4. :negative_squared_cross_mark: replace all vendor internal dependencies and add them into replace clause.
  5. :negative_squared_cross_mark: build and test the result. It must reproduce step 1 results.
  6. [ ] freeze the version using your VCS (git commit).
  7. [ ] migrate all internal dependencies into its dedicated modules externally.
  8. [ ] correct all the import statements to use the external modules created in step 7.
  9. [ ] build and test the result. It must reproduce step 1 results.
  10. [ ] freeze the version using your VCS (git commit).
  11. [ ] remove the replace clause.
  12. [ ] build and test the result. It must reproduce step 1 results.
  13. [ ] delete the vendor directory.
  14. [ ] freeze the version using your VCS (git commit).
  15. [ ] Treat yourself with a double cheese burger + cola [ ENDED ].

(Satish Anupindi) #9

Thank You Holloway!!! You’re a life-saver.


(Holloway) #10

Remember to do the long term migration when you have the resources. Please feel free to “mark as solved” for the most appropriate reply as solution.