Ultimate Go Service

Introduction

I teach a class called Ultimate Go. The class is three days long and teaches you the history, mechanics and semantics of the Go programming language. The idea is to teach you how to read code to the level that you can understand the behavior and impact your program is having on the machine. This helps you make better and more consistent engineering decisions.

At the same time, the class is very focused on teaching you how to optimize for correctness so the code you are writing is readable as a first priority. After three days in the class, I do believe you are a better developer. Not just a better Go developer, but a better developer in general. This is regardless of how much experience you had in Go or programming before you walked in the room.

The Problem

Here is the problem, Ultimate Go alone doesn’t prepare you to sit down to write a production service, cli tool, or background application. It can’t because there are other layers of foundational knowledge you need to learn. Ultimate Go is teaching you the first of what I believe are five layers of engineering and software development skills required in order to write production software. The reality of this has been weighing down on me.

What are these five layers of software development I just mentioned?

Layer 1: Language Mechanics, Semantics, Design, Debugging

Types, Syntax, Data Semantics, Composition, Error Handling, Tooling

Layer 2: Application Development

Architecture, Structure, Configuration, Naming, Refactoring, Mental Models

Layer 3: Observability

Logging, Metrics, Tracing

Layer 4: Deployment

Docker, Docker-Compose, Cloud

Layer 5: Source Control / Continuous Integration

Travis, Jenkins, Source Control

Each layer could use a full two day class. My future goal is to teach layers two through four in another three day class called Ultimate Go Service by writing, refactoring and walking through the code in this project.

Feeling Overwhelmed

I know developers who individually know every one of those layers I mentioned incredibly well. I would not say developers knowing all five layers well is the norm. I’m thinking about the philosophical and technical aspects of these layers on a daily basis. Recently all my thinking got me to the point where I started to feel overwhelmed.

My friend Nick Stogner told me:

“I think there is so much noise out there about these topics and strong opinions on the “right” way that I understand what you’re saying about being overwhelmed.”

So how do we as an industry comprehensively teach developers all of these layers of software development? How do we do this for individual companies with different constraints and needs? I don’t feel that I have the best answers yet.

What I’m Struggling With Today

Here are questions I believe need to be answered before you start writing production level applications. I’ve been trying to develop answers that I can teach to on these subjects. Obviously there are more questions but these are some I want solid answers for today.

How many solid answers do you have?

Q: How much knowledge of the production environment is required to be known by product developers?

When I started this journey I was hoping very little to nothing. But I have come to realize that if you don’t understand how the production system is architected and how code is managed and deployed, you can’t make the best engineering decisions. All you’re doing is causing pain for the operation developers who are now responsible for the mess you created.

Note: Yes, I consider anyone in operations a developer and I think we all need to do this.

Q: Where is the line in the sand for what product developers are responsible for and what the operation developers are responsible for?

I think there needs to be a lot of discussion about what is reasonable and practical for everyone involved. The less blurry the line is and the more empathy everyone has for each other, the better. There are some simple answers like, product developers should only write logs to stdout. Then again, maybe Docker containers are the handoff point and they define some of these touch points. I think product and operation developers need to define this line as far in advance as possible and appreciate each others role.

Q: Should the development of services for things like metrics and tracing be the responsibility of operation developers?

I think this is exactly the type of development operations could be responsible for. I think this also ties the product and operation developers closer together since they work together on the same code base.

Q: What is the purpose of the logs for the application?

This is a big question and takes more than a few sentences to answer. The bare minimum answer must identify a few things. Do the logs need to be structured or not? Are there different logging levels? How much informational data needs to be logged without creating more noise than signal? How should errors be logged? Where in the code does or can logging take place?

Q: How does the application get its configuration and is the application allowed to be restarted if configuration changes?

This is another big question and really affects operations. This question can’t be made by the product developers in isolation. Every service needs to be unified in how configuration is handled. One of the biggest problems with configuration is how hidden it can become in a code base. Sometimes to the point where you have no idea what all the configuration options are.

Q: How is the application going to isolate one request from the other in the logs? Does the application need function level tracing?

It’s critical to generate a unique ID for any request or task that is performed in a service. If you can’t filter logs by an individual request or task they are essentially useless. These trace ID’s should also be annotated in tracing data as well if you are generating function level traces. You might be surprised how many projects are not doing any of this today in their services.

Conclusion

If you have solid answers and examples for any of these questions I’m interested in hearing them. Maybe you can blog about it or even give a talk. Your experience, insights and opinions are needed. If you are feeling shaky don’t panic, I think most of us are shaky on this stuff.

My goal over the next few months is to finish the code and documentation for this new class. I’m not doing this alone and if you are interested in helping let me know. There is more to do and lots of documentation to write. The project has really been designed to serve as a starter-kit and I believe this is a great place today to start if you want to build services in Go.

If you are feeling overwhelmed today like me, my friend Misty reminded of something on Twitter I told her a couple years ago:

https://twitter.com/mistymadonna/status/1005228613778604032

Figure 1