I somehow ended up doing a talk at #jsconfeu this weekend on "make" of all things, submitted after a night of partying with all the wrong people who thought it would be a hilarious idea.
The long and short of this is that I work on a team of cross-platform developers on a project comprised mostly of Erlang and C. Make is the natural choice for this and indeed we have a common make file across our projects found at github/id3as/erl-mk.
Our core codebase is shiny diamonds, polished diamonds; It's stable and relatively robust, and we string together a lot of it with the programmer equivalent of duct tape (bash scripts). Make is for building, Bash is for task execution and this separation works very well for us.
Bringing in "yet another build system" just for JS would wind everybody up something chronic, especially when the build systems available in the JS world are confused about whether they are task runners or build scripts and the heinous mess of either JSON or JS streaming code found in this environment is pretty off-putting.
So yeah, we use Make because it's there and because it's good at building things; We use Bash because it's there and it's good at running things.
Also Make is just plain fun, it's such a quirky thing to work with how could you not enjoy it? (If you're working in JS you already know what it's like to work with something quirky and the two are therefore a match made in heaven)
All the examples shown in this entry assume you're in the directory with a file called "Makefile"
So lesson number one in the land of Make, is what happens in the following file when we run "make"?
one=$(two) two=$(three) three=hello world all: @echo $(one)
Well obviously we print out "hello world" to the console, why is that? Because anything assigned with '=' is a recursive variable, and they'll be recursively expanded at the point of use. (So in this case "one" isn't used until we echo it, at which point both two and three have values and we get a good result).
This is important because it impacts how we use expressions and functions later on in the Makefile.
A makefile is comprised of variables (as we see above) and recipes (which are a combination of targets, pre-requisites and some instructions to generate those targets)
out/file.txt: in/file.txt cp in/file.txt out/file.txt
The above is not idiomatic, but serves to highlight that a target is usually a file or directory and the pre-req is often a file/directory to a file/directory too. The instructions in the recipe simply instruct Make how to generate that target from that input.
Because we've stated our targets and pre-reqs in terms of files, Make can check the timestamps of these artifacts and only run the command if the pre-req is newer than the target.
Given this Makefile, what stands out?
all: out/pinkie.txt out/rainbow.txt @echo > /dev/null out/pinkie.txt: in/pinkie.txt out cp in/pinkie.txt out/pinkie.txt out/rainbow.txt: in/rainbow.txt out cp in/rainbow.txt out/rainbow.txt out: mkdir -p out
Well we're repeating ourselves a lot for what is a simple file copy. Make gives us "pattern rules" to help with this. Targets can be defined in terms of "patterns" and then we match on the pattern rather than an exact.
all: out/pinkie.txt out/rainbow.txt @echo > /dev/null out/%.txt: in/%.txt out cp $< $@ out: mkdir -p out
Woah, what is going on here? Well, the percentage symbol is the pattern we're matching on, and we're still invoking that target for both out/pinkie.txt and out/rainbow.txt. Make then gives us automatic variables to work with; There are a good dozen of these available, but in the above example we're using
Of course, we're still manually entering out/pinkie.txt and in/pinkie.txt and that's sub-optimal so let's sort that out too.
Various functions are available to us in Makefiles, and what we want to do is generate a list of outputs to invoke our targets with.
The only thing we have to work with is a directory full of inputs, so let's go and find all of those first
INPUTS := $(wildcard in/*.txt)
When the value of INPUTS is expanded, it'll contain a list of files matching the pattern, in this case that's the value "in/pinkie.txt in/rainbow.txt"
Our desired outputs actually have the same name at these except they're in a different directory, that's okay because we can call another function to replace all the 'in's n the INPUTS with 'out's
OUTPUTS := $(patsubst in/%,out/%, $(INPUTS))
Putting all of this together, our Makefile now looks like this
INPUTS := $(wildcard in/*.txt) OUTPUTS := $(patsubst in/%,out/%, $(INPUTS)) all: $(OUTPUTS) @echo > /dev/null out/%.txt: in/%.txt out cp $< $@ out: mkdir -p out
This is much tidier and this forms the basis of many a Makefile.
Referring to the example found on Github here: robashton/jsconf2014
We have a folder layout that looks like this
/ site/ index.html widgets/ pinkie/ img/ various-images.png index.styl index.coffee rainbow/ img/ various-images.png index.styl index.coffee celestia/ img/ various-images.png index.styl index.coffee
What we want to do is generate
site/widgets.js site/widgets.css img/*
From the widget folders, and we want to do so as efficiently as possible and in a way which means that Make has a good chance of not doing repeat work.
I like to start with my desired outputs and work out what I need to get there, in this case, ignoring the images this means
widgets.js <- cat widgets/*/*.js <- coffee -c widgets/*/*.coffee
widgets.css <- cat widgets/*/*.css <- stylus widgets/*/*.styl
(This does mean that requires won't work in Styl, but it doesn't require a lot of imagination to stretch this makefile to only run on index.styl and declare other styl files as dependencies of it)
So, first we need to bundle up our inputs
COFFEEFILES := $(wildcard widgets/*/*.coffee) STYLUSFILES := $(wildcard widgets/*/*.styl)
And generate our lists of outputs
OUTPUTJSFILES := $(patsubst %.coffee,%.js, $(COFFEEFILES)) OUTPUTCSSFILES := $(patsubst %.styl,%.css, $(STYLUSFILES))
Then our code path is quite simply
site/widgets.js: $(OUTPUTJSFILES) cat $^ > site/widgets.js widgets/%.js: widgets/%.coffee coffee -c $<
And our CSS path is quite simple too
site/widgets.css: $(OUTPUTCSSFILES) cat $^ > site/widgets.css widgets/%.css: widgets/%.styl $(STYLUSFILES) stylus $<
Images are a little more complicated because we're cheating and flattening the structure (everything from each /img folder goes directly in the output /img folder, so name clashes could happen. In the real world we'd probably stick things in a per widget folder or more likely generate a sprite map.)
SOURCEIMAGES := $(wildcard widgets/*/img/*.png) COPIEDIMAGES := $(addsuffix _lastcopied, $(SOURCEIMAGES))
What's this _lastcopied business? We'll see
widgets/%_lastcopied: widgets/% | site/img/ cp widgets/$* site/img/ @touch $@
We use a dummy file which we "touch" every time we copy an image, this is because in this example we lose the relationship between the input and output file. (This isn't necessary, it's just a demo to show you could do this). Make can then compare timestamps between that _lastcopied file and the image itself to determine whether it needs copying.
Our make all instruction now simply looks like this
all: site/widgets.js site/widgets.css $(COPIEDIMAGES) @echo > /dev/null
For bonus points, we can run make with
And all of this can be done in parallel!!. Most of the time it'll only be compiling a couple of coffee files anyway and will be super fast because Make doesn't like to do extra work.
So we've written an awesome makefile that generates spritemaps, compiles various languages into JS (coffee/JSX/whatever), and then executes browserify against this and perhaps even scans node_modules for templates/stylesheets/images/etc (Well why not eh?)
It'd be a shame not to use these conventions across our projects, but copying a Makefile into all those projects seems like an awful idea because you don't want all the projects to be the same but "slightly different".
How about using wget?
Our makefile can look something like this
export WIDGET_DIR=baubles common.mk: @wget -nv -O $@ 'https://raw.github.com/robashton/jsconfeu2014/master/src/_assets/common.mk' || rm -f $@ -include common.mk
Note the "export", we can see how this is used in common.mk over here
WIDGET_DIR ?= widgets .DEFAULT_GOAL=all
Nice - we can configure the makefile if we decide our widgets should be called baubles, for bonus points we then execute "all" by default, so just typing "Make" will result in this file being downloaded and then the project being compiled. Mega wins.
Note: While it is seemingly nice to use a package manager, for widget type designs like this, it's enough to just stick them into Github and recursively download the repos and either execute their make files directly or if they haven't got one, re-execute our make file in the context of that directory. (That's what we do in Erlang instead of using Rebar, but projects that have rebar or their own makefile work fine then too because we're just calling Make anyway).
I have no opinion on whether your team should do it one way or another.
I think most of the Grunt/Gulp whatevers and examples of usages of those I've seen in the wild get it pretty hilariously wrong because they tread a weird line between acting as task runners and build systems. What's even worse is when you see Makefiles that look simply like this
all: build-js build-css build-js: browserify -t brfs src/app.js > site/app.js build-css: stylus src/style.styl > site/style.css etc: blah .PHONY build-js build-css etc
For plain old task running, just stick things in npm scripts, for build processes use a build tool. In our projects this means Bash or Make but whatever works. Make's power is in generating dependency trees and only evaluating targets that are out of date (and then parallelising them). If this isn't being used then it's a bit pointless to be using Make.
It goes the other way, if we're spending a pile of time trying to set up dependency structures in whatever task runner is the current hotness, perhaps looking backwards to proven technology might be a sensible notion.
I hope this was useful if you've not considered make before or have considered make and were put off, it's a fine piece of software full of magic incantations and hilariously clearly added afterthoughts.
2018 © Rob Ashton. ALL Rights Reserved.