Hi, I hope this is not a stupid question. I am making an interpreter in Go for a while. In it’s core build it has very few dependencies. I use build tags to get an executable with more functionalities, which also means more dependencies. For example:
# creates a version with just few dependencies (like peg-parser, terminal library, etc)
go build
# creates a executable with much more functionalities and a lot more dependencies
go build -tags "http,bcrypt,psql,html,json"
# more examples with again totally different deps
go build -tags "gtk,sqlite"
go build -tags "raylib,sxml"
It makes absolutely no sense to require all those (and many more) dependencies from a user that
just wants to play with the core of interpreter. And if for example wants to do webdev, it makes no sense to require the instalation of gtk, raylib and similar unrelated libraries. Without go modules, you choose what you want at build time, and you needed to “go get” the dependencies you needed.
I want to upgrade the project to go.mod but I don’t know how. I found no way to define modules dependant on build tags or solve this in some other way. If I run go mod tidy it and downloads all possible modules that you could compile the executable with. I guess I could programmatically generate go.mod at compile time, but I suspect there must be some way to solve this already as this is probably not the only use case for non-static list of dependencies.
How big is the difference in terms of download time or volume between the minimal and the full set of dependencies? Given that the dependencies are downloaded once and then cached locally, users might accept a few more seconds of waiting for dependencies to download.
If you can split the various extra functionality into separate modules, you might be able to make use of lazy module loading, which was added with Go 1.17.
The difference is really big, because Rye (the language) can have all sorts of optional bindings, that can also be huge, hard to compile packages, that a specific user will never need. Like GTK, Raylib, Cayley (graph database) to name just few. If I caricature, it’s like you would need to install any possible program for your operating system, even though you will need just email and a browser
The problem is not just time, but also platform specific things. Concrete example, I develop it on ubuntu where I had all those specific go packages, but when a colleague on Mac tried to build it he had real problems installing the correct GTK, Raylib (game engine), which he had no interest in, since he just wanted to try and make FizzBuzz example with it.
Lazy module loading sounds promising, any kind of dynamic module loading determined at compile time would probably help.
How do people use Go.mod if they have for example platform specific dependencies?
I am not sure if I understand all aspects of the problem, but it seems that all the really huge parts are non-Go artifacts (GTK, Raylib, Cayley), while their respective Go bindings are probably not large at all (correct me if I am wrong).
A user would not have to install any of GTK, Raylib, or Cayley if they compile Rye without any of these extensions, correct? They would only have to wait for the Go command to download a few unnecessary binding packages that take some space in the module cache but would not go in the way.
Maybe you want to run a test to see how much Go code is actually downloaded. Run go clean -modcache to clear the local module cache, then run go mod tidy on your main module’s go.mod file, and then run something like du -sm $(go env GOPATH)/pkg/mod/cache to get the size (on disk) of the downloaded files. If this size seems too large for comfort, then splitting the project into separate modules and making use of lazy loading might help. (Disclaimer: I did not yet inspect the lazy loading behavior in detail. My assumption purely relies on the document I linked earlier.)
Hm … I haven’t thought of that you could just install the go bindings without the non-go artifacts if you don’t need them. This could be acceptable at the moment, although I would still much prefer tighter initial instalation with just few go-get’s.
I will test how much is actually downloaded for a full install right now. Thank you for the tip!
In the long run, language is supposed to get more and more bindings, so requiring to install all, even just Go parts from all of them is not a hindrance from the start.
I think I will need to make some build scripts and generate go.mod based on user input before compilation and loose just the simple go build -flags "pgsql,httpd" … I really liked that simplicity. And if I generate the go.mod on my own before compile time, I wouldn’t like to reinvent the wheel.
Come to think about it, it’s strange that go mod tidy doesn’t handle -flags in a same way as go build does?
Thanks you for the comment on Rye … Using Go to write it was a really good choice that keeps on giving, from all the platforms Go runs on, simple cross compilation, all the libraries, and Go is a very nice / non-risky language, so if you have some “hot” code, there is no problem in just rewriting that part in Go and calling it. Which wouldn’t be something you could expect from a higher level programmer to do in c/c++ (for example).
That’s a valid point. Indeed, I found an issue in golang/go about the use of build tags in go.mod:
However, the discussion does not seem to include the question about download times, it’s more about not having to add a build tag to every compile manually.
A solution discussed there is to split the code into separate modules and use a Makefile to compose the final app as needed.
