Updated on February 28th, 2017

Prelude

This post is part of a series of posts designed to make you think about your own design philosophy on different topics. If you haven’t read these posts yet, please do so first:

Develop Your Design Philosophy
Design Philosophy On Packaging

Introduction

Package Oriented Design allows a developer to identify where a package belongs inside a Go project and the design guidelines the package must respect. It defines what a Go project is and how a Go project is structured. Finally, it improves communication between team members and promotes clean package design and project architecture that is discussable.

Package oriented design is NOT bound to a single project structure, but states that a project structure is paramount to applying guidelines for good package design. Moving forward, I will present one possible project structure and the guidelines to follow based on the design philosophies presented earlier.

Project Structure

I believe that every company should establish a single Kit project and then multiple Application projects for the different sets of programs that get deployed together.

Kit Projects

Think of the Kit project as a company’s standard library, so there should only be one. The packages that belong to the Kit project need to be designed with the highest levels of portability in mind. These packages should be usable across multiple Application projects and provide a very specific but foundational domain of functionality. To this end, the Kit project is not allowed to have a vendor folder. If any of packages are dependent on 3rd party packages, they must always build against the latest version of those dependences.

A typical Kit project might look like this:

Listing 1

github.com/ardanlabs/kit
├── CONTRIBUTORS
├── LICENSE
├── README.md
├── cfg/
├── examples/
├── log/
├── pool/
├── tcp/
├── timezone/
├── udp/
└── web/

Note: There is nothing wrong with breaking each of these Kit packages into their own repository. I don’t do this because it creates more work managing all these packages. Vendoring tools that exist today let you cherry pick the packages you need to vendor from a repository. This feature allows you to manage a single repository for Kit packages.

Application Projects

Application projects contain the set of programs that get deployed together. The set of programs can include services, cli tooling and background programs. Each Application project is bound to a single repo that contains all the source code for that project, including all the source code for the 3rd party dependencies. How many Application projects you need is up to you, but always take a less is more approach.

Each Application project contains three root level folders. These are cmd/, internal/ and vendor/. There is also a platform/ folder inside of the internal/ folder, which has different design constraints from the other packages that live inside of internal/.

A typical Application project might look like this:

Listing 2

github.com/servi-io/api
├── cmd/
│   ├── servi/
│   │   ├── cmdupdate/
│   │   ├── cmdquery/
│   │   └── servi.go
│   └── servid/
│       ├── routes/
│       │   └── handlers/
│       ├── tests/
│       └── servid.go
├── internal/
│   ├── attachments/
│   ├── locations/
│   ├── orders/
│   │   ├── customers/
│   │   ├── items/
│   │   ├── tags/
│   │   └── orders.go
│   ├── registrations/
│   └── platform/
│       ├── crypto/
│       ├── mongo/
│       └── json/
└── vendor/
    ├── github.com/
    │   ├── ardanlabs/
    │   ├── golang/
    │   ├── prometheus/
    └── golang.org/

vendor/

Good documentation for the vendor/ folder can be found in this Gopher Academy post by Daniel Theophanes. For the purpose of this post, all the source code for 3rd party packages need to be vendored (or copied) into the vendor/ folder. This includes packages that will be used from the company Kit project. Consider packages from the Kit project as 3rd party packages.

cmd/

All the programs this project owns belongs inside the cmd/ folder. The folders under cmd/ are always named for each program that will be built. Use the letter d at the end of a program folder to denote it as a daemon. Each folder has a matching source code file that contains the main package.

internal/

Packages that need to be imported by multiple programs within the project belong inside the internal/ folder. One benefit of using the name internal/ is that the project gets an extra level of protection from the compiler. No package outside of this project can import packages from inside of internal/. These packages are therefore internal to this project only.

internal/platform/

Packages that are foundational but specific to the project belong in the internal/platform/ folder. These would be packages that provide support for things like databases, authentication or even marshaling.

Validate Package Design

An important aspect of package oriented design is the ability to validate the design of packages. This is possible because of the guidelines that are associated with a package based on its location inside the project. There are seven validation steps that will help you identify design problems.

Validate the location of a package.

• Kit
  • Packages that provide foundational support for the different Application projects that exist.
  • logging, configuration or web functionality.
• cmd/
  • Packages that provide support for a specific program that is being built.
  • startup, shutdown and configuration.
• internal/
  • Packages that provide support for the different programs the project owns.
  • CRUD, services or business logic.
• internal/platform/
  • Packages that provide internal foundational support for the project..
  • database, authentication or marshaling.

Validate the dependency choices.

• All
  • Validate the cost/benefit of each dependency.
  • Question imports for the sake of sharing existing types.
  • Question imports to others packages at the same level.
  • If a package wants to import another package at the same level:
    • Question the current design choices of these packages.
    • If reasonable, move the package inside the source tree for the package that wants to import it.
    • Use the source tree to show the dependency relationships.
• internal/
  • Packages from these locations CAN’T be imported:
    • cmd/
• internal/platform/
  • Packages from these locations CAN’T be imported:
    • cmd/
    • internal/

Validate the policies being imposed.

• Kit, internal/platform/
  • NOT allowed to set policy about any application concerns.
  • NOT allowed to log, but access to trace information must be decoupled.
  • Configuration and runtime changes must be decoupled.
  • Retrieving metric and telemetry values must be decoupled.
• cmd/, internal/
  • Allowed to set policy about any application concerns.
  • Allowed to log and handle configuration natively.

Validate how data is accepted/returned.

• All
  • Validate the consistent use of value/pointer semantics for a given type.
  • When using an interface type to accept a value, the focus must be on the behavior that is required and not the value itself.
  • If behavior is not required, use a concrete type.
  • When reasonable, use an existing type before declaring a new one.
  • Question types from dependencies that leak into the exported API.
    • An existing type may no longer be reasonable to use.

Validate how errors are handled.

• All
  • Handling an error means:
    • The error has been logged.
    • The application is back to 100% integrity.
    • The current error is not reported any longer.
• Kit
  • NOT allowed to panic an application.
  • NOT allowed to wrap errors.
  • Return only root cause error values.
• cmd/
  • Allowed to panic an application.
  • Wrap errors with context if not being handled.
  • Majority of handling errors happen here.
• internal/
  • NOT allowed to panic an application.
  • Wrap errors with context if not being handled.
  • Minority of handling errors happen here.
• internal/platform/
  • NOT allowed to panic an application.
  • NOT allowed to wrap errors.
  • Return only root cause error values.

Validate testing.

• cmd/
  • Allowed to use 3rd party testing packages.
  • Can have a test folder for tests.
  • Focus more on integration than unit testing.
• kit/, internal/, internal/platform/
  • Stick to the testing package in go.
  • Test files belong inside the package.
  • Focus more on unit than integration testing.

Validate recovering panics.

• cmd/
  • Can recover any panic.
  • Only if system can be returned to 100% integrity.
• kit/, internal/, internal/platform/
  • Can not recover from panics unless:
    • Goroutine is owned by the package.
    • Can provide an event to the app about the panic.

Quick Example

Here is a quick example of how we can review the Application project to understand how the project is put together and validate the dependency choices for each package.

• cmd/
  • This Application project builds two programs: servi and servid.
    • servid is a web service.
    • servi is a cli tool.
  • None of the packages inside of servid/ can import any packages from inside servi/.
    • The routes package CAN’T import the cmdupdate package.
  • The routes package does import the handlers package.
• internal/
  • This project has four root level internal packages.
    • attachments, locations, orders and registrations.
  • This project has three internal/platform/ packages.
    • crypto, mongo and sg
  • The four root level packages are not allowed to import each other.
    • They are at the same level.
    • attachments can’t import any other internal/ only package.
  • The orders package has three packages declared inside of it.
    • customers, items and tags.
    • From within the internal/ folder, only orders can import these packages.
    • These three packages can’t import each other.
  • Any package inside of the internal/ folder can import from internal/platform/.
    • The attachments package can import mongo.
    • The internal/platform/ packages can import each other.
      • The crypto can import the mongo.

Conclusion

Package Oriented Design fosters conversations and review to make sure packages maintain the best possible purpose, usability and portability. This drives clean package design for any package across the entire project. For package oriented design to be effective, you need strong rules about project structure. The project structure I have shared has been developed over the past three years and has been working effectively on multiple projects. Other project structures could be just as effective and I expect over time to continue to refactor my project structure and guidelines.

My goal in everything I teach is to get you to start thinking about what you are doing and why. To have you start asking questions and validate everything you are doing. I hope you start to think about package oriented design and begin to formalize how you structure projects and design packages for the projects you work on.