GNU Make is something I've always considered to be the necessary evil of C projects. Writing a Makefile with a call or two to GCC is relatively easy, but as a project gets bigger so do the caveats to the build behaviour.
Like much in the world of software development, there is a Catch 22 on knowing what you want to achieve and understanding enough to get useful search results. The seemingly archaic syntax of make (
$(@F)) doesn't make searching for help any easier. Admittedly, neither has my reluctance to "RTFM".
I recently decided it was time to take Make more seriously. Rather than just sit down and read the documentation from start to finish I figured a small case study would be more beneficial. I chose to look at the build system of ChibiOS, recalling that it was quite compact. I hoped this would make it easier to understand.
My focus was on the make files used by the RT-STM32F407-DISCOVERY example project. This was examined on the ChibiOS
stable_16.1x branch, at commit
The build system used in ChibiOS has two key parts:
- A per project Makefile, kept along with the project source code
- The rules.mk file, buried in the RTOS' support files.
The build system also uses supplementary
.mk files to bring in source files and include directories for separate components. The importing of which is handled within the main project's Makefile.
It's possible to create and assign values to variables in make. In fact, it's hard to avoid, and ChibiOS uses variables liberally. However, variable assignment behaves a little differently compared to C or similar languages. Understanding these differences is good a place to start..
Variable Assignment in Make
There are two distinct operators available to assign a value to a variable in make;
:=. They behave differently to each other in relation to the point when the variable is actually set. To understand the difference its helpful to know that the make operation has two distinct phases:
- Reads in all the variables, targets etc.
- Starts determining what is required to be built.
Also of note is that to reference a variable (substitute its value) it must be wrapped between the parentheses of
$(), otherwise it is treated as being a string.
Deferred Expansion (
= the expansion of the variable assignment is "deferred" until after the first phase completes. After the first phase, make is aware of all the variables specified in the makefile(s). If there are references to other variables in the value being assigned, make will expand them using their final (post phase one) value. The following example:
X = $(Y) Y = "hello"
will see the assignment of
X deferred until the after the first phase, at which point make is aware that
Y has also been assigned a value.
Reading the variable with
$(X) will give
Immediate Expansion (
The immediate expansion of variables can be requested by using the
:= operator. References to other variables are expanded during the first phase, at the moment of the assignment.
This is similar to as you would expect
= to work in C or Python. Unlike in these languages, referencing a variable which has not been assigned results in an empty string - not an error.
The following will result in later references to
X := $(Y) Y = "hello"
+= operator appends to a variable. Its behaviour depends on how the variable was previously assigned.
When the variable was not previously assigned, or it was assigned with
= then make will use deferred expansion.
The following is an example of this, with a later reference to
X += "One" X += $(Y) Y = "Two"
If the variable was assigned previously using immediate expansion, then appending to it will also be immediate. In the case below, referencing
$(X) will give
X := "One" X += $(Y) Y = "Two"
For more information consult the following sections of the documentation:
Back to ChibiOS, the per project Makefile handles pretty much anything the user might want to configure. Such as setting the:
- Toolchain (compiler, linker etc)
- Build options
- Source files
Nothing too interesting happens in this file - it's all just assignment to various variables.
For instance, the
ASMSRC variables are set in this file and respectively list the C and assembly source files. The
INCDIR variable lists the various directories that need to be searched for header files.
External, reusable modules (including the RTOS code itself) are added to the build via four steps:
- Define a variable with the path to the module (that can be referenced by the module's
- Include the module's
- Add the module's sources variable to
- Add the module's include directory variable to
For example the ChibiOS Hardware Abstraction Layer (HAL) is included to the build by:
CHIBIOSto the path to the repository
The required effort to add a module to ChibiOS could be reduced if each
*.mk file appended to
INCDIR directly. This would allow steps 3 and 4 to be removed. I suspect the current method was chosen since being more explicit, it is easier to debug. Since it also does not force the use of specific variables (
CSRC) it allows third party modules to be RTOS independent.
The last action of the Makefile is to
rules.mk file. The
include directive requests that make load a specified makefile. In this case it happens to be the one in which the real magic happens.
make - An Overview
Makefiles work by using rules. The rules ChibiOS uses to generate the output files are found in
rules.mk. Each consists of:
- a target - typically the name of the file you want to create
- prerequisites - the (optional) dependencies which must exist in order to build the target file
- recipe - the command to be executed to build the target
The syntax for a rule is:
target : dependencies recipe
If a rule has prerequisites specified, make will firstly ensure they can be found. If they cannot, make will try and invoke the appropriate rule to build them first. This also happens to the prerequisites of prerequisites.
Once all dependencies for a target have been found, make will determine if the target itself needs to be built. If the target file doesn't exist or if the timestamp of any dependency is newer than the target's timestamp, make will execute the rule's recipe.
There is a list of standard targets, which the make documentation recommends should be provisioned for. The idea being the same commands issued for different projects will have equivalent results.
One of these standard targets is
all - which should "compile the entire program". This is typically what we want to do with ChibiOS.
To invoke make with the goal of building the target
all, the following command can be issued:
If make is invoked without arguments it attempts to build the default goal. Since
all is the first target listed in either the Makefile or rules.mk, make determines it to be the default goal. Thus the following command has the same result as
rules.mk the target
all doesn't have a recipe, just the following list of dependencies:
all: PRE_MAKE_ALL_RULE_HOOK $(OBJS) $(OUTFILES)
Looking closer at
OUTFILES we see it is the following list of files:
OUTFILES := $(BUILDDIR)/$(PROJECT).elf \ $(BUILDDIR)/$(PROJECT).hex \ $(BUILDDIR)/$(PROJECT).bin \ $(BUILDDIR)/$(PROJECT).dmp \ $(BUILDDIR)/$(PROJECT).list
- .elf - The complete image in elf format, created after compiling and linking the various source files. The output of the
filecommand on this is
ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped. See below for more information on how it's built.
- .hex - The .elf converted to Intel Hex format. Generated by running
arm-none-eabi-objcopy -O ihexon the .elf.
- .bin - The .elf converted to binary format. Generated by running
arm-none-eabi-objcopy -O binaryon the .elf.
- .dmp - Symbol table. Generated by running
arm-none-eabi-objdump -x --symson the .elf.
- .list - Source intermixed with disassembly. Generated by running
arm-none-eabi-objdump -Son the .elf.
1. The .elf Rule
OUTFILES comprised of several different targets, the .elf file is what is loaded onto the board using GDB, so it will be my main focus.
The following is the rule used to build the project specific .elf file:
$(BUILDDIR)/$(PROJECT).elf: $(OBJS) $(LDSCRIPT) ifeq ($(USE_VERBOSE_COMPILE),yes) @echo $(LD) $(OBJS) $(LDFLAGS) $(LIBS) -o $@ else @echo Linking $@ @$(LD) $(OBJS) $(LDFLAGS) $(LIBS) -o $@ endif
LDSCRIPT are the necessary dependencies to build the .elf file.
In the case where the dependencies are met, the recipe dictates that the defined linker (aliased to
LD) is called to combine all the object files and libraries into the .elf.
The intended output of the linker is
$@. This is an automatic variable. When expanded, it becomes the current target (value left of the
:). In this case the current target
$(BUILDDIR)/$(PROJECT).elf is expanded to
endif block is an example of a conditional directive. The variable
USE_VERBOSE_COMPILE is used by ChibiOS to control the logging of additional information, with increased output to
stdout when set to
Typically makefiles print (aka echo) each line before executing it. Appending a
@ symbol in front of a line requests that echoing it be suppressed.
For example the following two lines:
echo hello world 1 @echo hello world 2
would produce the output:
echo hello world 1 hello world 1 hello world 2
When compiling with low verbosity, the only output from this recipe is "Linking build/ch.elf" while in verbose form the entire command is printed.
If not all of the files listed in
OBJS are available, make will attempt to build them before the
$(OBJS): | $(BUILDDIR) $(OBJDIR) $(LSTDIR)
There are two things to note in this line. Firstly, as we will soon see this is not the only time the value of
OBJS appears as a rule's target. This is allowed as it is only adding extra dependencies, not specifying the recipe.
Secondly, everything to the right of the
| symbol is an order-only prerequisite. For the requirements of this type of prerequisite to be met, the file just has to exist. The timestamp comparison between an order-only prerequisite and target is not made. This is often used for output directories, as is the case here.
OBJS itself comprises of various
.o files, which can be compiled from various sources. Some of these sources may be
.s but I'm more interested in the list of
.c files originally listed in Makefile as
CSRC. These are copied to the variable
TCSRC where the
T prefix denotes these files are to be compiled to the Thumb instruction set.
TCSRC += $(CSRC) TSRC := $(TCSRC) $(TCPPSRC) TCOBJS := $(addprefix $(OBJDIR)/, $(notdir $(TCSRC:.c=.o))) OBJS := $(ASMXOBJS) $(ASMOBJS) $(ACOBJS) $(TCOBJS) $(ACPPOBJS) $(TCPPOBJS)
There is a lot going on in the assignment of
TCOBJS := $(addprefix $(OBJDIR)/, $(notdir $(TCSRC:.c=.o)))
Working from the most nested parentheses outwards:
This is using a substitution reference. When expanded, any file in
TCSRC which ends in
.c will have its extension changed to
notdir is in fact a function - functions in make have a call syntax similar to variable referencing.
notdir is used to remove the directory path from files.
$(addprefix $(OBJDIR)/, $(notdir $(TCSRC:.c=.o)))
The last step uses the
addprefix function to prepend the desired output directory onto the front of each file.
There is a unique rule for each batch of object files, since each is intended to be compiled slightly differently. For example,
TCOBJS are to be compiled to the Thumb instruction set, while
ACOBJS are to be compiled to the ARM instruction set. Make has to be aware of the specific target filename so it can determine the appropriate recipe to use.
$(TCOBJS) : $(OBJDIR)/%.o : %.c Makefile ifeq ($(USE_VERBOSE_COMPILE),yes) @echo $(CC) -c $(CFLAGS) $(TOPT) -I. $(IINCDIR) $< -o $@ else @echo Compiling $(<F) @$(CC) -c $(CFLAGS) $(TOPT) -I. $(IINCDIR) $< -o $@ endif
This comes in the form:
targets: target-pattern: prereq-patterns
% character is used to match text, and can be used between an optional prefix and suffix. The text which it matches is referred to as the stem.
targets are matched against the pattern in
target-pattern, and the matching stem replaces the instance of
prereq-patterns. Everything in
prereq-patterns becomes the prerequisites of this target.
Continuing using the previous example file, and expanding the variables:
$(TCOBJS) : $(OBJDIR)/%.o : %.c Makefile
build/obj/chsys.o : build/obj/%.o : %.c Makefile
chsys is the stem of the pattern rule, and after pattern matching this looks like:
build/obj/chsys.o : chsys.c Makefile
The recipe for the
TCOBJS target invokes the compiler to create the target object file. Two useful automatic variables are used here:
$<- Name of the first prerequisite (e.g.
$@- Name of the rule target (e.g.
Specifying the Makefile as a prerequisite is a useful trick. This will ultimately force a rebuild of the project if the Makefile is edited (e.g. if the user adds new source files or changes compiler options).
Along the way, the
.c prerequisites to this rule (e.g.
chsys.c) have been stripped of their full path. This would ordinarily cause problems for make and gcc, since they need to be able to locate the files. ChibiOS sets make's
VPATH variable to get around this problem.
VPATH is a list of directories that make will search to try and find a prerequisite or target file. Helpfully, if the filename is located in a
VPATH location the appropriate path is appended to the prerequisite filename.
VPATH to include the various directories of the source files:
SRCPATHS := $(sort $(dir $(ASMXSRC)) $(dir $(ASMSRC)) $(dir $(ASRC)) $(dir $(TSRC))) VPATH = $(SRCPATHS)
Header Files as Prerequisites
We seen above that
Makefile was added as a prerequisite of a rule to ensure that if the Makefile was updated the project would be rebuilt.
However, none of the rules we looked at considered header files. Header files are awkward to account for, mainly because nobody wants to have to manually track them as being the prerequisites of source files.
Thankfully, flags can be passed to GCC to allow it to take care of this for us:
CFLAGS += -MD -MP -MF .dep/$(@F).d
These flags (
-MF) are options intended for the preprocessor. They request the generation of dependency files that are compatible with make. Essentially, this requests the preprocessor generate the output files that contain rules listing the header file dependencies.
For example, the dependency file for chsys.o.d looks like:
build/obj/chsys.o: ../../../os/rt/src/chsys.c ../../../os/rt/include/ch.h \ ../../../os/rt/ports/ARMCMx/compilers/GCC/chtypes.h \ /usr/lib/gcc/arm-none-eabi/4.8/include/stddef.h \ /usr/lib/gcc/arm-none-eabi/4.8/include/stdint.h \ /usr/include/newlib/stdint.h \ # # etc... #
@F is another automatic variable. It is a short equivalent of
$(notdir $@) i.e. the basename of the target file.
The appending to
CFLAGS is a good example of a deferred expansion. If
$(@F) were expanded immediately the result would be an empty variable. Since
CFLAGS was initially set using
CFLAGS = the assignment will be deferred, resulting in the expansion to the appropriate target filename each time.
The GCC generated rules are loaded into make with the following command in
-include $(shell mkdir .dep 2>/dev/null) $(wildcard .dep/*)
Considering the individual parts:
-- ignore any errors from this command.
include- read in a make file (in this case multiple).
$(shell mkdir .dep 2>/dev/null)- the
shellfunction performs "command expansion". It spawns a new shell, invokes the specified command and expands to the output of the command. Its use here seems almost like an abuse of the shell function. This mechanism is used to ensure the
.depdirectory exists at the point
includeis called. The executed command uses
stderrto hide any logged errors. After some testing, the
shellcommand appears to ignore any output from
stderr, so the redirection appears redundant.
wildcard- This function is expanded to any files which match the pattern
.dep/*- i.e. all the contents of the
.depdirectory. I don't believe this function is necessary here since the documentation for
includestates: "filenames can contain shell file name patterns".
The ChibiOS build system does a lot in relatively few lines of make. I cannot currently think of any significant functionality that is missing. It is definitely an example I will look to in the future when writing a makefile.
After taking time to study it, I have to admit the documentation for GNU Make is pretty good. However, it is likely easier to navigate when using it as a reference for a known command than it would be when searching for an appropriate command.
In the descriptions above I have not always followed the path to build dependencies that make (probably) would. Since the same dependency may occur for more than one target, I have tried to follow a more logical path. E.g.
OBJS is a dependency of
all and a dependency of
.elf. Following the latter seemed like a more appropriate choice.