Series Index
Why and What
Projects, Dependencies and Gopls
Minimal Version Selection
Mirrors, Checksums and Athens
Gopls Improvements
Vendoring
Introduction
Every dependency management solution has to solve the problem of picking a version of a dependency. Many of the version selection algorithms that exist today attempt to identify the “latest greatest” version of any dependency. This makes sense if you believe semantic versioning will be applied correctly and the social contract will be respected. In these cases, the “latest greatest” version of a dependency should be the most stable and secure version and should have backwards compatibility with earlier versions. At least in the same major version dependency tree.
Go decided to take a different approach and Russ Cox has spent a lot of time and energy writing and talking about the Go team’s approach to version selection which is called Minimal Version Selection or MVS. Essentially, the Go team believes that MVS provides Go programs with the best opportunity for durable and reproducible builds over the long haul. I recommend reading this post to understand why the Go team believes this.
In this post, I will do my best to explain the MVS semantics and show a real-world example of Go and the MVS algorithm in action.
MVS Semantics
Naming Go’s selection algorithm “minimal version selection” is a bit of a misnomer, but once you learn how it works you will see the name comes really close. As I stated before, many selection algorithms select the “latest greatest” version of a dependency. I like to think of MVS as an algorithm that selects the “latest non-greatest” version. It’s not that MVS can’t select the “latest greatest”, it’s just that if the “latest greatest” is not required by any dependency in the project, that version isn’t needed.
To better understand this, let’s create a situation where several modules (A, B and C) are depending on the same module (D) but each require a different version.
Figure 1
Figure 1 shows how modules A, B and C all independently require module D and each require a different version of the module.
If I start a project that requires module A, then in order to build the code I also require module D. There could be many versions of module D to choose from. For example, imagine module D represents the logrus module from sirupsen. I can ask Go to provide me a list of all the versions that have been tagged for module D.
Listing 2
$ go list -m -versions github.com/sirupsen/logrus
github.com/sirupsen/logrus v0.1.0 v0.1.1 v0.2.0
v0.3.0 v0.4.0 v0.4.1 v0.5.0 v0.5.1 v0.6.0 v0.6.1
v0.6.2 v0.6.3 v0.6.4 v0.6.5 v0.6.6 v0.7.0 v0.7.1
v0.7.2 v0.7.3 v0.8.0 v0.8.1 v0.8.2 v0.8.3 v0.8.4
v0.8.5 v0.8.6 v0.8.7 v0.9.0 v0.10.0 v0.11.0 v0.11.1
v0.11.2 v0.11.3 v0.11.4 v0.11.5 v1.0.0 v1.0.1 v1.0.3
v1.0.4 v1.0.5 v1.0.6 v1.1.0 v1.1.1 v1.2.0 v1.3.0
v1.4.0 v1.4.1 v1.4.2
Listing 2 shows all the versions that exist for module D which shows the “latest greatest“ version to be v1.4.2.
Which version of module D should be selected for the project? There are really two choices. The first choice is to select the “latest greatest” version (in this line of major version 1 releases) which would be version v1.4.2. The second choice is to select the version that module A requires which is version v1.0.6.
A dependency tool like dep would select version v1.4.2 and work under the assumption of semantic versioning and the social contract being respected. However, for reasons defined by Russ in this post, Go is going to respect module A’s requirements and select version v1.0.6. Go is selecting the “minimal” version that is currently in the set of required versions for all the dependencies in the project that require the module. In other words, right now only module A requires module D and module A has specified it requires version v1.0.6, so that is the version of module D that will be selected.
What if I introduce new code that requires the project to import module B? Once module B is imported into the project, Go upgrades the version of module D for the project from v1.0.6 to v.1.2.0. Once again selecting the “minimal” version of module D that is currently in the set of required versions (v1.0.6 and v.1.2.0) for all the dependencies (modules A and B) in the project that require module D.
What if I introduce new code once more that requires the project to import module C? Then Go will choose the latest version (v1.3.2) from the set of required versions (v1.0.6, v1.2.0, v1.3.2). Note that version v1.3.2 is still a “minimal” version and not the “latest greatest” version of module D (v1.4.2).
Lastly, what if I remove the code I just added for module C? Go will lock the project into version v1.3.2 for module D. To downgrade back to version v1.2.0 would be a bigger change and Go knows version v1.3.2 works and is stable, so version v1.3.2 remains the “latest non-greatest” or “minimal” version of module D for the project. Plus, the module files only maintain a snapshot and are not a log. There is no information for historical undoing or downgrading.
This is why I like to think of MVS as an algorithm that picks the “latest non-greatest” version of a module. Hopefully you now understand why Russ chose the name “minimal” when naming the algorithm.
Example Project
With this foundation in place, I will put together a project so you can see Go and the MVS algorithm in action. In this project, module D will represent the logrus module and the project will directly depend on the rethinkdb-go (module A) and golib (module B) modules. The rethinkdb-go and golib modules directly depend on the logrus module and each require a different version that is not the “latest greatest” version of logrus.
Figure 2
Figure 2 shows the independent relationship between the three modules. To start, I will create the project, initialize modules, and then load VS Code.
Listing 2
$ cd $HOME
$ mkdir app
$ mkdir app/cmd
$ mkdir app/cmd/db
$ touch app/cmd/db/main.go
$ cd app
$ go mod init app
$ code .
Listing 2 shows all the commands to run. After running those commands, the following should appear in VS Code.
Figure 3
Figure 3 shows what the project structure and module file should contain. With this in place, it’s time to add code that will use the rethinkdb-go module.
Listing 3
https://play.golang.org/p/bc5I0Afxhvc
01 package main
02
03 import (
04 "context"
05 "log"
06
07 db "gopkg.in/rethinkdb/rethinkdb-go.v5"
08 )
09
10 func main() {
11 c, err := db.NewCluster([]db.Host{{Name: "localhost", Port: 3000}}, nil)
12 if err != nil {
13 log.Fatalln(err)
14 }
15
16 if _, err = c.Query(context.Background(), db.Query{}); err != nil {
17 log.Fatalln(err)
18 }
19 }
Listing 3 introduces major version 5 of the rethinkdb-go module. After adding and saving this code, Go finds, downloads and extracts the module, updating the go.mod and go.sum files.
Listing 4
01 module app
02
03 go 1.13
04
05 require gopkg.in/rethinkdb/rethinkdb-go.v5 v5.0.1
Listing 4 shows the go.mod file requiring the rethinkdb-go module as a direct dependency selecting version v5.0.1, which is the “latest greatest” version of that module.
Listing 5
...
github.com/sirupsen/logrus v1.0.6 h1:hcP1GmhGigz/O7h1WVUM5KklBp1JoNS9FggWKdj/j3s=
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
...
Listing 5 shows two lines from the go.sum file that introduces version v1.0.6 of the logrus module. At this point you can see that the MVS algorithm has selected the “minimal” version of the logrus module needed to satisfy the requirement specified by the rethinkdb-go module. Remember the “latest greatest” version of the logrus module is 1.4.2.
Note: The go.sum file should be considered an opaque reliability artifact and it shouldn’t be used to understand your dependencies. What I am doing above to determine versions is wrong and shortly I will show you the right way to determine what version is being used for your project.
Figure 4
Figure 4 shows a visual of which version of the logrus module Go will use to build the code in the project.
Next I will add code that introduces a dependency on the golib module.
Listing 6
https://play.golang.org/p/h23opcp5qd0
01 package main
02
03 import (
04 "context"
05 "log"
06
07 "github.com/Bhinneka/golib"
08 db "gopkg.in/rethinkdb/rethinkdb-go.v5"
09 )
10
11 func main() {
12 c, err := db.NewCluster([]db.Host{{Name: "localhost", Port: 3000}}, nil)
13 if err != nil {
14 log.Fatalln(err)
15 }
16
17 if _, err = c.Query(context.Background(), db.Query{}); err != nil {
18 log.Fatalln(err)
19 }
20
21 golib.CreateDBConnection("")
22 }
Listing 6 added lines 07 and 21 to the program. Once Go finds, downloads and extracts the golib module, the following changes appear in the go.mod file.
Listing 7
01 module app
02
03 go 1.13
04
05 require (
06 github.com/Bhinneka/golib v0.0.0-20191209103129-1dc569916cba
07 gopkg.in/rethinkdb/rethinkdb-go.v5 v5.0.1
08 )
Listing 7 shows the go.mod file has been modified to include the dependency on the golib module for the “latest greatest” version of that module, which happens to not have a semantic version tag.
Listing 8
...
github.com/sirupsen/logrus v1.0.6 h1:hcP1GmhGigz/O7h1WVUM5KklBp1JoNS9FggWKdj/j3s=
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.2.0 h1:juTguoYk5qI21pwyTXY3B3Y5cOTH3ZUyZCg1v/mihuo=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
...
Listing 8 shows four lines from the go.sum file that now include versions v1.0.6 and v1.2.0 of the logrus module. Seeing both versions listed in the go.sum file brings up two questions:
- Why are both versions listed in the
go.sum file? - Which version is going to be used when Go performs a build?
The reason both versions are listed in the go.sum file is better answered by Bryan Mills from the Go team.
“The go.sum file still includes the older version (1.0.6) because its transitive requirements may affect the selected versions of other modules. We really only need the checksum for the go.mod file, since that is what declares those transitive requirements, but we end up retaining the checksum for the source code too because go mod tidy is not as precise as it ought to be.”
golang.org/issue/33008
This still leaves the question of which version of the logrus module will be used when building the project. To correctly identify which modules and their version will be used don’t look at the go.sum file, but rather use the go list command.
Listing 9
$ go list -m all | grep logrus
github.com/sirupsen/logrus v1.2.0
Listing 9 shows that version v1.2.0 of the logrus module will be used when building the project. The -m flag directs go list to list modules instead of packages.
Looking at the module graph will provide more insight into the requirements the project has on the logrus module.
Listing 10
$ go mod graph | grep logrus
github.com/sirupsen/logrus@v1.2.0 github.com/pmezard/go-difflib@v1.0.0
github.com/sirupsen/logrus@v1.2.0 github.com/stretchr/objx@v0.1.1
github.com/sirupsen/logrus@v1.2.0 github.com/stretchr/testify@v1.2.2
github.com/sirupsen/logrus@v1.2.0 golang.org/x/crypto@v0.0.0-20180904163835-0709b304e793
github.com/sirupsen/logrus@v1.2.0 golang.org/x/sys@v0.0.0-20180905080454-ebe1bf3edb33
gopkg.in/rethinkdb/rethinkdb-go.v5@v5.0.1 github.com/sirupsen/logrus@v1.0.6
github.com/sirupsen/logrus@v1.2.0 github.com/konsorten/go-windows-terminal-sequences@v1.0.1
github.com/sirupsen/logrus@v1.2.0 github.com/davecgh/go-spew@v1.1.1
github.com/Bhinneka/golib@v0.0.0-20191209103129-1dc569916cba github.com/sirupsen/logrus@v1.2.0
github.com/prometheus/common@v0.2.0 github.com/sirupsen/logrus@v1.2.0
Listing 10 shows the relationships the logrus module has in the project. I will extract the lines that show the dependency requirements on logrus directly.
Listing 11
gopkg.in/rethinkdb/rethinkdb-go.v5@v5.0.1 github.com/sirupsen/logrus@v1.0.6
github.com/Bhinneka/golib@v0.0.0-20191209103129-1dc569916cba github.com/sirupsen/logrus@v1.2.0
github.com/prometheus/common@v0.2.0 github.com/sirupsen/logrus@v1.2.0
In listing 11, these lines show that three modules (rethinkdb-go, golib, common) all require the logrus module. Thanks to the go list command, I know that the minimal version required is version v1.2.0.
Figure 5
Figure 5 shows a visual of which version of the logrus module Go will now use to build the code in the project for all dependencies that require the logrus module.
Go Mod Tidy
Before you commit/push code back to the repo, run go mod tidy to make sure your module files are current and accurate. The code you’ve been building, running or testing locally will affect what Go decides at any time to update in the module files. Running go mod tidy will guarantee the project has an accurate and complete snapshot of what is needed and this will help others on your team and your CI/CD environments.
Listing 12
$ go mod tidy
go: finding github.com/Bhinneka/golib latest
go: finding github.com/bitly/go-hostpool latest
go: finding github.com/bmizerany/assert latest
Listing 12 shows the output from running go mod tidy. You can see two new dependencies show up in the output. This changes the module files.
Listing 13
01 module app
02
03 go 1.13
04
05 require (
06 github.com/Bhinneka/golib v0.0.0-20191209103129-1dc569916cba
07 github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932 // indirect
08 github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 // indirect
09 gopkg.in/rethinkdb/rethinkdb-go.v5 v5.0.1
10 )
Listing 13 shows that the go-hostpool and assert modules are listed as indirect modules required to build the project. They are being listed here because those projects are not currently compliant with modules. In other words, a go.mod file does not exist in the repo for any tagged version or the “latest greatest” version in master for these projects.
Why were these modules included after running go mod tidy? I can use the go mod why command to find out.
Listing 14
$ go mod why github.com/hailocab/go-hostpool
# github.com/hailocab/go-hostpool
app/cmd/db
gopkg.in/rethinkdb/rethinkdb-go.v5
github.com/hailocab/go-hostpool
------------------------------------------------
$ go mod why github.com/bmizerany/assert
# github.com/bmizerany/assert
app/cmd/db
gopkg.in/rethinkdb/rethinkdb-go.v5
github.com/hailocab/go-hostpool
github.com/hailocab/go-hostpool.test
github.com/bmizerany/assert
Listing 14 shows why these modules are indirectly required for the project. The rethinkdb-go module requires the go-hostpool module and the go-hostpool module requires the assert module.
Upgrading Dependencies
The project has three dependencies each requiring the logrus module where version v1.2.0 of the logrus module is currently being selected. At some point in the lifecycle of the project it will become important to upgrade the direct and indirect dependencies to make sure the required code for the project is current and can take advantage of new features, bug fixes and security patches. To apply upgrades, Go provides the go get command.
Before you run go get to upgrade the project’s dependencies, there are several options that need to be considered.
Upgrade Only Required Direct and Indirect Dependencies Using MVS
It’s my recommendation to start with this kind of upgrade until you learn more about your project and modules. This is the most conservative form of go get.
Listing 15
$ go get -t -d -v ./...
Listing 15 shows how to perform an upgrade that will only focus on required dependencies using the MVS algorithm. Here are the definitions for the flags.
-t flag: Consider modules needed to build tests.-d flag: Download the source code for each module but do not build or install them.-v flag: Provide a verbose output../... : Perform these operations across the entire source tree and only update dependencies that are required.
Running this command against the current project will cause nothing to change since the project is already up to date with the minimal versions required to build and test the project. That’s because I just ran go mod tidy and the project is new.
Upgrade Only Required Direct and Indirect Dependencies Using Latest Greatest
This kind of upgrade will raise the dependencies from “minimal” to “latest greatest” across the entire project. All that is required is adding the -u flag to the command line.
Listing 16
$ go get -u -t -d -v ./...
go: finding golang.org/x/net latest
go: finding golang.org/x/sys latest
go: finding github.com/hailocab/go-hostpool latest
go: finding golang.org/x/crypto latest
go: finding github.com/google/jsonapi latest
go: finding gopkg.in/bsm/ratelimit.v1 latest
go: finding github.com/Bhinneka/golib latest
Listing 16 shows the output from running the go get command with the -u flag. This output doesn’t tell the real story. What happens if I ask the go list command which version of the logrus module is now being used to build the project?
Listing 17
$ go list -m all | grep logrus
github.com/sirupsen/logrus v1.4.2
Listing 17 shows how the “latest greatest” version of logrus is now being selected. To put this selection in stone, a change was made to the go.mod file.
Listing 18
01 module app
02
03 go 1.13
04
05 require (
06 github.com/Bhinneka/golib v0.0.0-20191209103129-1dc569916cba
07 github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932 // indirect
08 github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 // indirect
09 github.com/cenkalti/backoff v2.2.1+incompatible // indirect
10 github.com/golang/protobuf v1.3.2 // indirect
11 github.com/jinzhu/gorm v1.9.11 // indirect
12 github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
13 github.com/sirupsen/logrus v1.4.2 // indirect
14 golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413 // indirect
15 golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 // indirect
16 golang.org/x/sys v0.0.0-20191210023423-ac6580df4449 // indirect
17 gopkg.in/rethinkdb/rethinkdb-go.v5 v5.0.1
18 )
Listing 18 shows on line 13 that version v1.4.2 is now the selected version for the logrus module in the project. This line in the module file is what is respected by Go when building the project. Even if code is removed that changes the dependency on the logrus module, version v1.4.2 is now locked in stone for this project. Remember, to downgrade would be a bigger change than leaving version v.1.4.2 moving forward.
What changes can be seen in the go.sum file?
Listing 19
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.2.0 h1:juTguoYk5qI21pwyTXY3B3Y5cOTH3ZUyZCg1v/mihuo=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
Listing 19 shows how all three versions of logrus are now represented in the go.sum file. As explained by Bryan above, this is because transitive requirements may affect the selected versions of other modules.
Figure 6
Figure 6 shows a visual of which version of the logrus module Go will now use to build the code in the project for all dependencies that require the logrus module.
Upgrade All Direct and Indirect Dependencies Using Latest Greatest
You can substitute the ./... option for all to upgrade and include all the direct and indirect dependencies including the ones you don’t need to build the project.
Listing 20
$ go get -u -t -d -v all
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0
go: finding github.com/bitly/go-hostpool latest
go: finding github.com/denisenkom/go-mssqldb latest
go: finding github.com/hailocab/go-hostpool latest
go: finding gopkg.in/bsm/ratelimit.v1 latest
go: finding github.com/google/jsonapi latest
go: finding golang.org/x/net latest
go: finding github.com/Bhinneka/golib latest
go: finding golang.org/x/crypto latest
go: finding gopkg.in/tomb.v1 latest
go: finding github.com/bmizerany/assert latest
go: finding github.com/erikstmartin/go-testdb latest
go: finding gopkg.in/check.v1 latest
go: finding golang.org/x/sys latest
go: finding github.com/golang-sql/civil latest
Listing 20 shows how many more dependencies are now found, downloaded and extracted for the project.
Listing 21
Added to Module File
cloud.google.com/go v0.49.0 // indirect
github.com/denisenkom/go-mssqldb v0.0.0-20191128021309-1d7a30a10f73 // indirect
github.com/google/go-cmp v0.3.1 // indirect
github.com/jinzhu/now v1.1.1 // indirect
github.com/lib/pq v1.2.0 // indirect
github.com/mattn/go-sqlite3 v2.0.1+incompatible // indirect
github.com/onsi/ginkgo v1.10.3 // indirect
github.com/onsi/gomega v1.7.1 // indirect
github.com/stretchr/objx v0.2.0 // indirect
google.golang.org/appengine v1.6.5 // indirect
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
gopkg.in/yaml.v2 v2.2.7 // indirect
Removed from Module File
github.com/golang/protobuf v1.3.2 // indirect
Listing 21 shows the changes to the go.mod file. Many more modules were added and one module was removed.
Note: If you are vendoring, the go mod vendor command strips out test files from the vendor folder.
As a general guideline, don’t use the all option or the -u flag when upgrading dependencies using go get for your projects. Stick to just the modules you need and to using the MVS algorithm to select those modules and their versions. Manually override specific module versions when necessary. Manual overrides can be done by manually editing the go.mod file which I will show you in a future post.
Resetting Dependencies
If at any time you are not comfortable with the modules and the versions being selected, you can always reset the selections by removing the module files and running go mod tidy again. This is more of an option when the project is young and things are not stable. Once a project is stable and released, I would hesitate to reset the dependencies. As I mentioned above, module versions over time may get set and you want durable and reproducible builds over the long haul.
Listing 22
$ rm go.*
$ go mod init <module name>
$ go mod tidy
Listing 22 shows the commands you can run to allow MVS to perform all the selections again from scratch. I have been doing this throughout the writing of this post to reset the project and provide the listings for the post.
Conclusion
In this post I explained the MVS semantics and showed a real-world example of Go and the MVS algorithm in action. I also showcased some of the Go commands that can provide you information if you get stuck or run into unknown issues. There are edge cases that you can run into as you add more and more dependencies to your projects. This is because the Go ecosystem is 10 years old and it will take more time for all of the existing projects to become module compliant.
In future posts I will talk about using dependencies of different major versions in the same project and how to manually retrieve and lock down specific versions of a dependency. For now, I hope you can trust modules and the Go tooling even more and that you have a clearer idea of how MVS is selecting versions over time. If you run into any problems, there is a community of people available on Gopher Slack in the #module group ready and willing to help.