For those of you who are like me, trying to learn the Mac and Linux operating systems, Golang programming and deployment constructs all at the same time, I feel your pain. I have been building a Go application for a couple of months on my Mac and it was time to deploy the code on a local Ubuntu server. I was having a really tough time and it was turning into a disaster. Like always, I kept telling myself, I must be doing something wrong.

Well, I was, big-time. Let me put it this way. After spending 2 days reorganizing repositories and code, I finally figured out how the Go tool works. Now I can update my development environment, deploy, build, install and update my Go applications on any machine with one simple call to go get.

I am sure there are several ways you can organize code that will work with the Go tool. What I am going to present is working for me and I wish I knew all of this when I started. There are a 1000 ways to skin a cat and we all have our favorite way, this one is now mine.

Everything we do in Go should be based on what the Go tool can do. Going off the reservation will get you in trouble, I am living proof. One special command we are going to focus on is get. The get command will download code, build and install packages and produce an executable binary if you are building a program. It is very smart and can read your code to find dependences that it will download, built and install as well. That is, if everything is structured and set correctly.

This document explains all the things the Go tool can do:

http://golang.org/cmd/go/

But I want to concentrate on this section:

http://golang.org/cmd/go/#hdr-Remote_import_path_syntax

Even as I read this document now I have a hard time understanding what it is trying to tell me.

I use Github but you don’t have to. The Go tool supports Github, Mercurial, Subversion and Bazaar out of the box. If you are not using any of these systems for version control, there are ways to give the Go tool the information it needs to support your version control system.

When I say the Go tool supports these version control systems it’s a bit vague right now, so let’s jump into this. Everything starts with a repository, so let’s look at the ones I have for my project. I have two accounts in Github. One is called goinggo which contains all my shared and reusable repositories. The other is ArdanStudios which contains my private repositories.

Here is the goinggo repository:

Choosing a name for your repository is really important. The name of the repository is going to be used later to reference the code. Having a good name makes things easier for everyone. You can see I have 2 of my repositories listed. Both of these repositories contain reusable code and are referenced in my project.

When I first put my repository together for the GoingGo.net website, I pull all these projects under a single repository. This ended up being a very bad idea. It is best to organize your code in different repositories that may or may not be needed depending on the project. As an example, I will never need the newsearch code for any project I am building. That was an application I built for an article I wrote. When I had that code in the same repository as the utilities code, and I reference just one code file from utilities, the Go tool still pulled down everything in that repository. Not Good.

Let's look at the utilities repository:



You will notice the repository has a folder with a major version number. There is a good amount of debate and research going on in the Go community about package management and versioning. This is a complicated problem with many twists and turns. There is a document that is being worked on by several people who are doing the research and leading the discussion. Here is that document:

https://docs.google.com/document/d/1_IJTRD6dDQvyCfyim4KJexq8ZrKUPvUd3GMSA8cw8A4/edit#heading=h.w4quaql3iduf

Since we can't wait for the final outcome to this debate, I have chosen to structure my repository with a major version number and with the idea that future versions of the Go tool will provide better version control system support. I also imagine that we will eventually have a standard package manager that will provide all the support we need to make this manageable and flexible.

There is another reason for using a major version number. I don't want users to worry about making changes to the references of this package every time I have a new minor or patch release. Since minor versions and patches are not allowed to break existing interfaces, this reference to just the major version is enough. Obviously I must comply with this rule and give a personal guarantee.

If I want to work on version 2, which will not be compatible with version 1, I can create a new v2 folder and not affect those who rely on the code. This also gives others confidence that they can use and rely on my package, which is very important.

Many feel that trusting others blindly to not break compatibility is too risky. I can understand that position. One option to mitigate that risk is to take a copy of the packages you want to use and repo it yourself. Being able to do this depends on the license for the code so check that first.


You will also notice I have two branches on this code. A develop branch and a master branch. Thanks to the Git Flow tool this is really easy to setup.

This allows me to work in the develop branch, making changes to the code without affecting those who are using the master branch. People can use the develop branch if they wish and start testing new features and bug fixes. I can also get feedback before any final release. Once I feel the new version is stable, I can merge and push the final changes into master and tell the world the release is ready. It is good practice to tag or label your branches, especially in master after a release of code. This will allow access to previous releases if it is necessary.

In the case of Github, the Go tool is going to download and use master. If you are building a package for the masses, branching and versioning is something you want to work out in the beginning.

Let's look at the project code that is sitting in the Github repository under my private account:



You can see this repository is under my ArdanStudios account and I have a single source code file called main.go. You can also see some of the internal package folders that make up the project.

Let's look inside the mongo folder and view the mongo.go code file that is in there.


This single code file in the mongo folder provides connection support for accessing my MongoDB database.


What is important to see is the import references. Notice that the references to the goinggo utilities, mongo driver and even the internal helper package is all done with a full url qualifier.

This is going to allow the Go tool to download these packages for us when we are ready to build and install our program.

For all this to work, in both our development and production environments, the code must reside in physical folders that match these url paths.


The Projects and PublicPackages folders in my development environment are located under a parent folder called Go. Under each of these folders I have a special Go folder called src.

In order for the Go tool to find the source code for the packages we import, the code must reside underneath a folder called src. In my development environment, the GOPATH contains these two folder:

$HOME/Spaces/Go/PublicPackages

Notice the directories we add to the GOPATH variable point to the src folders. The Go tool assumes there is a src folder immediately following each folder listed in the GOPATH.

If you look at the folders underneath src you will see the entire url is laid out as sub-folders and then the code folders and files follow:


In my development environment I like having a folder called Projects and PublicPackages. Code in the PublicPackages folder is just that, public code that I don't own and I am using in my projects. I could keep a single GOPATH and put the public code and my project code under a single folder. There is nothing wrong with that. I just like separating the code that I own from the code I don't.

To build out the PublicPackages folder you must manually bring down each repository yourself using to Go tool. Let's say you wanted to use the GoingGo utilities code in your dev environment.

Here is what you do. Open a terminal session and run these commands:


I always run go env after I set the GOPATH to double check that it is properly set. In this example it should say GOPATH="/Users/bill/example".

When you run the go get command on the goinggo utilities repository, you will get the following message:


This is because there is nothing for the Go tool to build. It's ok because this package just provides utility code that will be built by each individual project. If you remember, I pointed out how there was a main.go file in the root of my project code. This is why. I want to make sure the Go tool finds the main source code file to build the application.

You can specify additional paths in the call to go get if the code you want to build is not in the root of the project. Something like this:


When this runs you will not get any warnings and a static library file for workpool will exist in the package folder. All of the code for the specified repository will still be downloaded. Adding the extra folders to the end of the repository url only tell the Go tool where to start building the code.

When we open the example folder we see the Go tool created the entire tree structure:


What is even better is we have a cloned Github repository. The image on the right shows the hidden files for the .git folder. Why is this important? Anytime we want to update the code we can run the Go tool again:


Using the -u option will perform an update to the local repository. Setting up the PublicPackages folder for your development environment will keep one version of all the packages you use in a single place under a single GOPATH folder. Minimizing the number of GOPATH folders in your development environment is always a good thing. You can also update any of the public code if you need to very quickly.

Next let's simulate a production build and installation using the Mongo Rules program. This is going to show you the real power of the Go tool when we structure our repositories and code correctly.

Before we can try this, we need to install the Bazaar program. Mongo Rules references the labix.org mgo driver. The mgo driver is being held in a Bazaar version control system and the Go tool can not download the code without it. This is a great example of a project that is getting code from multiple types of repositories and version control systems.

If you are running on a Mac or Windows use these links and follow the instructions:

[http://wiki.bazaar.canonical.com/MacOSXDownloads](/broken-link)
[http://wiki.bazaar.canonical.com/WindowsDownloads](/broken-link)

If you are running on Linux just run apt-get:


With Bazzar installed we can use the Go tool to download and install the Mongo Rules program in our simulated production environment.

** WAIT **   It is important that the GOBIN environment variable is not set. If this variable is set then the Go tool will attempt to install the Mongo Rules program in the location specified by the variable. This is not what we want. To clear the variable if it is set, issue this call:



Now run these commands:


After the Go tool is done we have a build and installed program that is ready to go. Look at the directory structure after the call:


How cool is this !!

With a single call to go get all the code, including the packages the code depends on is downloaded, built and then installed.

If you look at the bin folder you will see the executable binary for the Mongo Rules program that was built.

Inside the pkg folders we have the static library files that were produced when go get performed the build.

In the src folder you can see all the code, including the code from the labix.org website that was downloaded. The Go tool looked at all the public references in the code and downloaded and built everything it needed to create a final executable.

What is also really nice is that everything works within a single directory from our GOPATH.

GOPATH=$HOME/example

If you want to learn more about this program check out this article I wrote for Safari Books Online. It talks about how you can use MongoDB and Go to analyze data.

https://www.ardanlabs.com/blog/2013/07/analyze-data-with-mongodb-and-go.html

All of this knowledge came to light when I needed to build, deploy and install my program on a different OS and machine. I really struggled to create a build package that would install all these dependencies and put everything in the right place. Little did I know a few days ago that the Go tool does all this for you. You just need to know how it works. Hopefully now, you do as well.