I have a server that can make use of 2 different packages without having to change anything in the code, just the import path.
When I import package A (and comment out package B), the result size is 26MB and takes 3 minutes to compile.
When I import package B (and comment out package A), the result size is 44MB and takes 2 minutes to compile.
I would expect the compilation with package B to take at least 5 minutes or at least not less than with package A.
Another reason for compilation time-differences could be build caching.
You can do go env or more specifically go env GOCACHE to find the directory where go keeps build caches. Also read go help cache.
If you have the time-command (normally part of your shell), you can time the difference between …
… a non-cached build: time go build -a
… a cached build: time go build
And of course, go build does more than just compiling (and linking). It can also download files if they are not yet in your go env GOPATH.