Series Index

Why and What
Projects, Dependencies and Gopls
Minimal Version Selection
Mirrors, Checksums and Athens
Gopls Improvements
Vendoring

Introduction

Modules is the system integrated into Go to provide support for dependency management. This means modules touch just about anything related to working with source code, including editor support. To provide editors with support for modules (and for other reasons), the Go team built a service named gopls which implements the language server protocol (LSP). LSP was originally developed by Microsoft for VS Code and has become an open standard. The idea of the protocol is to provide editors with support for language features such as auto complete, go to definition, and find all references.

When you are using modules and VS Code, hitting save in your editor will no longer run the go build command directly. Now what happens is that a request is sent to gopls, and gopls runs the appropriate Go commands and associated API’s to provide editor feedback and support. Gopls can also send information to the editor without the need of a request. There are times where it appears the editor is lagging or out of sync with a code change due to the nature of LSP or the inherent latencies in running Go commands. The Go team is working hard to reach a v1.0 of gopls to handle these edge cases so you can have the smoothest possible editor experience.

In this post, I will walk through the basic workflow for adding and removing dependencies from within your projects. This post is using the VS Code editor, version 0.2.0 of gopls and version 1.13.3 of Go.

Module Cache

To help speed up builds and keep dependency changes in your projects up to date quickly, Go maintains a cache of all the modules that it has ever downloaded on your local machine. That cache can be found at $GOPATH/pkg. If you don’t have a GOPATH setup, the default GOPATH is at $HOME/go.

Note: There is a proposal to provide an environmental variable to allow a user to control the location of the module cache. $GOPATH/pkg will be the default if not changed.

Listing 1

$HOME/code/go/pkg
$ ls -l
total 0
drwxr-xr-x  11 bill  staff  352 Oct 16 15:53 mod
drwxr-xr-x   3 bill  staff   96 Oct  3 16:49 sumdb

Listing 1 shows what my current $GOPATH/pkg folder looks like. You can see there are two folders, mod and sumdb. If you look inside the mod folder you can learn more about the module cache layout.

Listing 2

$HOME/code/go/pkg
$ ls -l mod/
total 0
drwxr-xr-x   5 bill  staff   160 Oct  7 10:37 cache
drwxr-xr-x   3 bill  staff    96 Oct  3 16:55 contrib.go.opencensus.io
drwxr-xr-x  40 bill  staff  1280 Oct 16 15:53 github.com
dr-x------  26 bill  staff   832 Oct  3 16:50 go.opencensus.io@v0.22.1
drwxr-xr-x   3 bill  staff    96 Oct  3 16:56 golang.org
drwxr-xr-x   4 bill  staff   128 Oct  7 10:37 google.golang.org
drwxr-xr-x   7 bill  staff   224 Oct 16 15:53 gopkg.in
drwxr-xr-x   7 bill  staff   224 Oct 16 15:53 k8s.io
drwxr-xr-x   5 bill  staff   160 Oct 16 15:53 sigs.k8s.io

Listing 2 shows the top level structure of my current module cache. You can see how the first part of the URL associated with the module’s name is used as a top level folder in the module cache. If I navigate into github.com/ardanlabs, I can show you two actual modules.

Listing 3

$HOME/code/go/pkg
$ ls -l mod/github.com/ardanlabs/
total 0
dr-x------  13 bill  staff  416 Oct  3 16:49 conf@v1.1.0
dr-x------  18 bill  staff  576 Oct 12 10:08 service@v0.0.0-20191008203700-49ed4b4f1088

Listing 3 shows two modules and their version that I am using from ArdanLabs. The first one is the conf module and the other module is associated with the service project that I use to teach kubernetes and services.

The gopls server also maintains a module cache that is kept in memory. The moment you start VS Code and you are in module mode, a gopls server is started to support that editor session. The internal gopls module cache is then synced with what is currently on disk. It’s this internal module cache that gopls uses to handle editor requests.

For this post, I am going to clear out my module cache before I begin so I have a clean working environment. I’m also going to setup my project before I start a VS Code editor. This will allow me to show you how to handle situations when the module you need hasn’t been downloaded yet to your local module cache or is updated in the gopls internal module cache.

Note: Clearing out your module cache is something you should never need to do in any normal workflow.

Listing 4

$ go clean -modcache

Listing 4 shows how to clear the local module cache on disk. The go clean command has been traditionally used to clean your local GOPATH working directory and GOPATH bin folder. Now with the new -modcache flag, the command can be used to clean the module cache.

Note: This command will not clear the internal cache of any running gopls instance.

New Project

I am going to start a new project outside of my GOPATH and in the process of writing code, I will walk through the basic workflows of adding and removing dependencies.

Listing 5

$ cd $HOME
$ mkdir service
$ cd service
$ mkdir cmd
$ mkdir cmd/sales-api
$ touch cmd/sales-api/main.go

Listing 5 shows commands to set up the working directory, create the initial project structure and add the main.go file.

The first step when working with modules is to initialize the root of your project’s source tree. This is done by using the go mod init command.

Listing 6

$ go mod init github.com/ardanlabs/service

Listing 6 shows the call to go mod init, passing the name of the module as a parameter. As discussed in the first post, the name of the module allows internal imports to be resolved inside the module. It’s idiomatic to name the module after the URL of the repo hosting the code. For this post, I will pretend this module will be associated with the service repo under Ardan Labs in Github.

Once the call to go mod init is complete, a go.mod file is created in the current working directory. This file will denote the root of the project.

Listing 7

01 module github.com/ardanlabs/service
02
03 go 1.13

Listing 7 shows the contents of the initial module file for this project. With that in place, the project is ready for coding.

Listing 8

$ code .

Listing 8 shows the command to launch an instance of VS Code. This will in turn start an instance of the gopls server to support this editor instance.

Figure 1

Figure 1 shows what the project in my VS Code editor looks like after running all of the commands. Just to make sure you are using the same settings as I am, I’ll list my current VS Code settings.

Listing 9

{
    // Important Settings
    "go.lintTool": "golint",
    "go.goroot": "/usr/local/go",
    "go.gopath": "/Users/bill/code/go",

    "go.useLanguageServer": true,
    "[go]": {
        "editor.snippetSuggestions": "none",
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": true
        }
    },
    "gopls": {
        "usePlaceholders": true,    // add parameter placeholders when completing a function
        "completeUnimported": true, // autocomplete unimported packages
        "deepCompletion": true,     // enable deep completion
    },
    "go.languageServerFlags": [
        "-rpc.trace", // for more detailed debug logging
    ],
}

Listing 9 shows my current VS Code settings. If you follow along and don’t see the same behavior, check these settings against yours. If you would like to see the current recommended VS Code settings they are here.

Application Code

I’m going to start with this initial set of code for the application.

Listing 10
https://play.golang.org/p/c8kGx7I9HJH

01 package main
02
03 func main() {
04     if err := run(); err != nil {
05         log.Println("error :", err)
06         os.Exit(1)
07     }
08 }
09
10 func run() error {
11     return nil
12 }

Listing 10 shows the first 12 lines of code I’m adding to main.go. It sets up the ability for the application to have a single point of exit and logging for any errors on start-up or shutdown. Once these 12 lines of code are saved to the file, the editor will automagically (thanks to gopls) include the imports required from the standard library.

Listing 11
https://play.golang.org/p/x3hBA6PuW3R

03 import (
04     "log"
05     "os"
06 )

Listing 11 shows the changes to the source code on lines 03 through 06 thanks to the editor integration with gopls.

Next, I will add support for configuration.

Listing 12
https://play.golang.org/p/4hFXLJj4yT_Z

17 func run() error {
18     var cfg struct {
19         Web struct {
20             APIHost         string        `conf:"default:0.0.0.0:3000"`
21             DebugHost       string        `conf:"default:0.0.0.0:4000"`
22             ReadTimeout     time.Duration `conf:"default:5s"`
23             WriteTimeout    time.Duration `conf:"default:5s"`
24             ShutdownTimeout time.Duration `conf:"default:5s"`
25         }
26     }
27
28     if err := conf.Parse(os.Args[1:], "SALES", &cfg); err != nil {
29         return fmt.Errorf("parsing config : %w", err)
30     }

Listing 12 shows the code that was added to the run function on lines 18 through 30 to support configuration. When this code is added to the source file and I hit save, the editor properly includes the fmt and time package to the set of imports. Unfortunately, since gopls doesn’t have any information about the conf package currently inside its internal module cache, gopls can’t direct the editor to add an import for conf or provide the editor with package information.

Figure 2

Figure 2 shows how the editor is making it clear that it can’t resolve any information related to the conf package.

Adding A Dependency

In order to resolve the import, the module that contains the conf package needs to be retrieved. One way this can be done is by adding the import to the top of the source code file and letting the editor and gopls do the work.

Listing 13

01 package main
02
03 import (
04     "fmt"
05     "log"
06     "os"
07     "time"
08
09     "github.com/ardanlabs/conf"
10 )

In listing 13, I add the import for the conf package on line 09. Once I hit save, the editor reaches out to gopls and then gopls finds, downloads and extracts the module for this package using the Go command and associated API’s. These calls also update the Go module files to reflect this change.

Listing 14

~/code/go/pkg/mod/github.com/ardanlabs
$ ls -l
total 0
drwxr-xr-x   3 bill  staff    96B Nov  8 16:02 .
drwxr-xr-x   3 bill  staff    96B Nov  8 16:02 ..
dr-x------  13 bill  staff   416B Nov  8 16:02 conf@v1.2.0

Listing 14 shows how the Go command did its job and downloaded the conf module using version 1.2.0. The code we need to resolve the import is now in my local module cache.

Figure 3

Figure 3 shows how the editor still can’t resolve information about the package. Why is the editor not able to resolve this information? Unfortunately, the gopls internal module cache is out of sync with the local module cache. The gopls server isn’t aware of the changes the Go command just made. Since gopls uses its internal cache, gopls can’t provide the editor with the information it needs.

Note: This shortcoming is currently being worked on and will be fixed in an upcoming release. You can track the issue here. (https://github.com/golang/go/issues/31999)

A quick way to get the gopls internal module cache back in sync with the local module cache is to reload the VS Code editor. This will restart the gopls server and reset its internal module cache. In VS Code, there is a special command called reload window to do just this.

Ctrl + Shift + P and run  > Reload Window

Figure 4

Figure 4 shows the dialog box that comes up in VS Code after using Ctrl + Shift + P and type reload window.

After running this quick command, any messages associated with the import will be resolved.

Transitive Dependencies

From the Go tooling’s point of view, all the code that is needed to build this application is now present in the local module cache. However, the conf package has a dependency on the Google go-cmp package for its tests.

Listing 15

module github.com/ardanlabs/conf

go 1.13

require github.com/google/go-cmp v0.3.1

Listing 15 shows the module file for version 1.2.0 of the conf module. You can see conf depends on version 0.3.1 of go-cmp. This module is not listed in the service’s module file because it would be redundant to do so. The Go tooling can follow the path of module files to get a complete picture of all the modules needed to build or test code.

At this point, this transitive module has not been found, downloaded and extracted to my local module cache yet. Since this module is not needed when building the code, the Go build tool hasn’t found the need to download it yet. If I run go mod tidy on the command line, then the Go tooling will take the time to bring the go-cmp module into my local cache.

Listing 16

$ go mod tidy
go: downloading github.com/google/go-cmp v0.3.1
go: extracting github.com/google/go-cmp v0.3.1

In listing 16 shows how the go-cmp module has been found, downloaded and extracted. This call to go mod tidy won’t change the module file for the project since this is not a direct dependency. It will update the go.sum file so there is a record of the module’s hash to maintain a durable and reproducible build. I will talk about the checksum database in a future post.

Listing 17

github.com/ardanlabs/conf v1.2.0 h1:2IntiqlEhRk+sYUbc8QAAZdZlpBWIzNoqILQvV6Jofo=
github.com/ardanlabs/conf v1.2.0/go.mod h1:ILsMo9dMqYzCxDjDXTiwMI0IgxOJd0MOiucbQY2wlJw=
github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=

Listing 17 shows what the checksum file looks like after running go mod tidy. There are two records for each module associated with the project.

Downloading Modules

If you are not ready to use a particular module inside your code base but want to download the module into your local module cache, one option is to manually add the module to the projects go.mod file and then run go mod tidy outside the editor.

Listing 18

01 module github.com/ardanlabs/service
02
03 go 1.13
04
05 require (
06     github.com/ardanlabs/conf v1.2.0
07     github.com/pkg/errors latest
08 )

In listing 18 you see how I manually added line 07 in the module file for the latest version of the errors module. The important part of manually adding the required module is using the latest tag. Once I run go mod tidy against this change, it will tell Go to find the latest version of the errors module and download it into my cache.

Listing 19

$HOME/service
$ go mod tidy
go: finding github.com/pkg/errors v0.8.1

Listing 19 shows how version 0.8.1 of the errors module was found, downloaded and extracted. Once the command is done running, the module is removed from the module file since the module is not being used by the project. However, the module is listed in the checksum file.

Listing 20

github.com/ardanlabs/conf v1.2.0 h1:2IntiqlEhRk+sYUbc8QAAZdZlpBWIzNoqILQvV6Jofo=
github.com/ardanlabs/conf v1.2.0/go.mod h1:ILsMo9dMqYzCxDjDXTiwMI0IgxOJd0MOiucbQY2wlJw=
github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=

Listing 20 shows how a hash for the module file for the errors module is now listed in the checksum file. It’s important to remember that the checksum file isn’t a canonical record of all the dependencies being used by the project. It can contain more modules and this is absolutely fine.

I like this approach to downloading new modules over using go get because go get can also attempt to upgrade dependencies in the dependency graph (direct and indirect) for the project if you’re not careful. It’s important to know when version upgrades are happening over just downloading a new module you want. In a future post, I will talk about using go get to update existing module dependencies.

Removing Dependencies

What happens if I decide I don’t want to use the conf package any longer? I can remove any code using the package.

Listing 21
https://play.golang.org/p/x3hBA6PuW3R

01 package main
02
03 import (
04     "log"
05     "os"
06 )
07
08 func main() {
09     if err := run(); err != nil {
10         log.Println("error :", err)
11         os.Exit(1)
12     }
13 }
14
15 func run() error {
16     return nil
17 }

Listing 21 shows the removal of the code referencing the conf package from the main function. Once I hit save, the editor removes the import for conf from the import set. However, the module file hasn’t been updated to reflect the change.

Listing 22

01 module github.com/ardanlabs/service
02
03 go 1.13
04
05 require github.com/ardanlabs/conf v1.1.0
06

Listing 22 shows that the conf package is still considered to be required. To fix this, I need to leave the editor and run go mod tidy once again.

Listing 23

$HOME/service
$ go mod tidy

Listing 23 shows the running of go mod tidy once more. This time there is no output. Once this command finishes, the module file is accurate again.

Listing 24

$HOME/services/go.mod

01 module github.com/ardanlabs/service
02
03 go 1.13
04

Listing 24 shows that the conf module has been removed from the module file. This time the go mod tidy command cleared out the checksum file and it will be empty. It’s important before you commit any changes into your VCS for your project to run go mod tidy and make sure your module files are accurate and consistent with the dependencies you are using.

Conclusion

In the near future, some of the workarounds I have shared like reloading the window will no longer be necessary. The Go team is aware of this and other shortcomings that exist today and they are actively working on fixing all of them. They do appreciate any and all feedback on the Go issue tracker so if you discover an issue please report it. No issue is too big or small. As a community let’s work with the Go team to resolve these remaining issues quickly.

One core feature being worked on now is the ability of gopls to watch the filesystem and see project changes for itself. This is going to help with gopls keeping its internal module cache in sync with the local module cache on disk. Once this is in place, the need to reload the window should go away. Plans are in the works as well to provide visual cues that work is being done in the background.

Overall I am happy with the current set of tooling and the reload window workaround. I hope you consider to start using modules if you’re not already. Modules are ready for use and the more projects that begin to use it, the better the Go ecosystem will be for everyone.