A Good Makefile for Go

I occasionally tweak my Makefiles to speed up my development process, this morning was one of those times and I decided to share the result with others.

To summarize, I use Go for building web servers and my expectation from a Makefile is as following:

High-level, simple commands. Such as; compile start stop watch, etc.
Managing project-specific environment variables. It should inclide .env file.
Development-mode that auto-compiles on change.
Development-mode that shows compile error without verbosity around it.
Project-specific GOPATH, so I can keep dependencies in vendor folder.
Simplified file watching. e.g make watch run="go test ./..."

And here is the typical directory layout I prefer:

.env
Makefile
main.go
bin/
src/
vendor/

Typing make command in this file structure gives following output:

 $  make

 Choose a command run in my-web-server:

 install   Install missing dependencies. Runs `go get` internally.
 start     Start in development mode. Auto-starts when code changes.
 stop      Stop development mode.
 compile   Compile the binary.
 watch     Run given command when code changes. e.g; make watch run="go test ./..."
 exec      Run given command, wrapped with custom GOPATH. e.g; make exec run="go test ./..."
 clean     Clean build files. Runs `go clean` internally.

Index of Contents

1. Step-by-step
2. Final Version

1. Step by step

Environment Variables

The very first thing we want from our Makefile to include the environment variables we define for our project. So, here is our line #1:

include .env

On top of the project-specific environment variables; we'll define some more; project name, Go folders/files, pid filepaths...

PROJECTNAME=$(shell basename "$(PWD)")

# Go related variables.
GOBASE=$(shell pwd)
GOPATH=$(GOBASE)/vendor:$(GOBASE):/home/azer/code/golang # You can remove or change the path after last colon.
GOBIN=$(GOBASE)/bin
GOFILES=$(wildcard *.go)

# Redirect error output to a file, so we can show it in development mode.
STDERR=/tmp/.$(PROJECTNAME)-stderr.txt

# PID file will store the server process id when it's running on development mode
PID=/tmp/.$(PROJECTNAME)-api-server.pid

# Make is verbose in Linux. Make it silent.
MAKEFLAGS += --silent

In the rest of the Makefile, we'll be using especially GOPATH variable heavily. All our commands should be wrapped with the project specific GOPATH, otherwise they won't work. This provides a clear isolation between our Go projects, and brings some complexity. To make things easier, we can add an exec command that takes executes any given command with custom GOPATH defined above.

## exec: Run given command, wrapped with custom GOPATH. e.g; make exec run="go test ./..."
exec:
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) $(run)

This is not high-level enough though. We should cover some common cases with simple commands, and only use exec if we're doing something not covered by the makefile.

Development-mode

Development-mode should;

Clean up the build cache
Compile the code
Run server on background
Repeat the steps above when code changes.

It sounds simple, but gets complicated quickly, as we'll run server and file watcher at same time. We need to make sure stopping properly before starting a new process, and also not to break common command-line behavior like stopping when Control-C or Control-D is pressed.

start:
	bash -c "trap 'make stop' EXIT; $(MAKE) compile start-server watch run='make compile start-server'"

stop: stop-server

Here is the problems above code solves:

Compiles and runs server on background.
Main process doesn't run on background. So we can interrupt when we want, using Control-C.
Stops background processes when the main process is interrupted. We need trap mainly for this.
Recompiles and restarts the server when code changes.

In the below sections I'll explain these commands in detail.

Compiling

compile command does more than just calling go compile in the background; it cleans up the error output and prints the simplified version.

Here is how making a breaking change looks like in the command-line:

compile:
	@-touch $(STDERR)
	@-rm $(STDERR)
	@-$(MAKE) -s go-compile 2> $(STDERR)
	@cat $(STDERR) | sed -e '1s/.*/\nError:\n/'  | sed 's/make\[.*/ /' | sed "/^/s/^/     /" 1>&2

Starting/stopping Server

start-server basically runs the binary it compiled in background, saving its PID to a temporary file. stop-server reads the PID and kills the process when needed.

start-server:
	@echo "  >  $(PROJECTNAME) is available at $(ADDR)"
	@-$(GOBIN)/$(PROJECTNAME) 2>&1 & echo $$! > $(PID)
	@cat $(PID) | sed "/^/s/^/  \>  PID: /"

stop-server:
	@-touch $(PID)
	@-kill `cat $(PID)` 2> /dev/null || true
	@-rm $(PID)

restart-server: stop-server start-server

Watching for Changes

We need a file watcher for watching for changes. I tried many of them and didn't feel satisfied, so ended up creating my own file watcher tool, yolo. Install it in your system by;

$  go get github.com/azer/yolo

Once it's installed, we can basically start watching changes in the project directory, excluding folders like vendor and bin. Here it is;

## watch: Run given command when code changes. e.g; make watch run="echo 'hey'"
watch:
	@yolo -i . -e vendor -e bin -c $(run)

Now we got a watch command that watches for changes recursively in the project directory excluding vendor directory. We can simply pass any run command we want. For example, start command basically calls make compile start-server when code changes:

make watch run="make compile start-server"

We can use it for running tests, or checking if there is any race condition automatically. Environment variables will be set for the execution, so you don't have to worry about GOPATH at all:

make watch run="go test ./..."

A nice thing about Yolo is its web interface. If you enable it, you can see the output of your command in a web interface instantly. All you need is to pass -a option to enable it:

yolo -i . -e vendor -e bin -c "go run foobar.go" -a localhost:9001

Then you can open localhost:9001 in your browser and start seeing results in your browser instantly:

Installing Dependencies

As we make changes in the code, we'd like missing dependencies to be downloaded before compiling. install command will do that job for us;

install: go-get

We'll automate calling install on file change before compiling, so dependencies will get installed automatically. If you'd like to install a dependency manually, you can run;

make install get="github.com/foo/bar"

Internally, this command will be converted to;

$ GOPATH=~/my-web-server GOBIN=~/my-web-server/bin go get github.com/foo/bar

How does it work though ? See the next section where we actually add the Go commands that we use for implementing the higher level commands.

Go Commands

As we want to set the GOPATH to the project directory to simplify dependency management which is still not solved officially in Go ecosystem, we need to wrap all Go commands in the Makefile.

go-compile: go-clean go-get go-build

go-build:
	@echo "  >  Building binary..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go build -o $(GOBIN)/$(PROJECTNAME) $(GOFILES)

go-generate:
	@echo "  >  Generating dependency files..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go generate $(generate)

go-get:
	@echo "  >  Checking if there is any missing dependencies..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go get $(get)

go-install:
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go install $(GOFILES)

go-clean:
	@echo "  >  Cleaning build cache"
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go clean

Help

Finally, we need a help command to see the overview available commands. We can auto-generate nicely formatted help output using sed and column commands. See below:

help: Makefile
	@echo " Choose a command run in "$(PROJECTNAME)":"
	@sed -n 's/^##//p' $< | column -t -s ':' |  sed -e 's/^/ /'

The command below basically scans the Makefile for lines starting with ## and outputs them. So, you can simply comment the commands that you defined, and those comments will be used by the help command.

If we add some comments like below:

## install: Install missing dependencies. Runs `go get` internally.
install: go-get

## start: Start in development mode. Auto-starts when code changes.
start:

## stop: Stop development mode.
stop: stop-server

We'll get:

 $  make help

 Choose a command run in my-web-server:

 install   Install missing dependencies. Runs `go get` internally.
 start     Start in development mode. Auto-starts when code changes.
 stop      Stop development mode.

Final Version

Here is the all combined & final version of what I've shared above. It's the exact copy from a new project that I've started this morning:

include .env

PROJECTNAME=$(shell basename "$(PWD)")

# Go related variables.
GOBASE=$(shell pwd)
GOPATH="$(GOBASE)/vendor:$(GOBASE)"
GOBIN=$(GOBASE)/bin
GOFILES=$(wildcard *.go)

# Redirect error output to a file, so we can show it in development mode.
STDERR=/tmp/.$(PROJECTNAME)-stderr.txt

# PID file will keep the process id of the server
PID=/tmp/.$(PROJECTNAME).pid

# Make is verbose in Linux. Make it silent.
MAKEFLAGS += --silent

## install: Install missing dependencies. Runs `go get` internally. e.g; make install get=github.com/foo/bar
install: go-get

## start: Start in development mode. Auto-starts when code changes.
start:
    bash -c "trap 'make stop' EXIT; $(MAKE) compile start-server watch run='make compile start-server'"

## stop: Stop development mode.
stop: stop-server

start-server: stop-server
	@echo "  >  $(PROJECTNAME) is available at $(ADDR)"
	@-$(GOBIN)/$(PROJECTNAME) 2>&1 & echo $$! > $(PID)
	@cat $(PID) | sed "/^/s/^/  \>  PID: /"

stop-server:
	@-touch $(PID)
	@-kill `cat $(PID)` 2> /dev/null || true
	@-rm $(PID)

## watch: Run given command when code changes. e.g; make watch run="echo 'hey'"
watch:
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) yolo -i . -e vendor -e bin -c "$(run)"

restart-server: stop-server start-server

## compile: Compile the binary.
compile:
	@-touch $(STDERR)
	@-rm $(STDERR)
	@-$(MAKE) -s go-compile 2> $(STDERR)
	@cat $(STDERR) | sed -e '1s/.*/\nError:\n/'  | sed 's/make\[.*/ /' | sed "/^/s/^/     /" 1>&2

## exec: Run given command, wrapped with custom GOPATH. e.g; make exec run="go test ./..."
exec:
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) $(run)

## clean: Clean build files. Runs `go clean` internally.
clean:
	@(MAKEFILE) go-clean

go-compile: go-clean go-get go-build

go-build:
	@echo "  >  Building binary..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go build -o $(GOBIN)/$(PROJECTNAME) $(GOFILES)

go-generate:
	@echo "  >  Generating dependency files..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go generate $(generate)

go-get:
	@echo "  >  Checking if there is any missing dependencies..."
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go get $(get)

go-install:
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go install $(GOFILES)

go-clean:
	@echo "  >  Cleaning build cache"
	@GOPATH=$(GOPATH) GOBIN=$(GOBIN) go clean

.PHONY: help
all: help
help: Makefile
	@echo
	@echo " Choose a command run in "$(PROJECTNAME)":"
	@echo
	@sed -n 's/^##//p' $< | column -t -s ':' |  sed -e 's/^/ /'
	@echo

That was it! If you have any questions, thoughts or some recommendations to make it better, shoot me e-mail!

Cheers.

Back