Calculating Actual Build Dependencies

During development, when you make a change your build system should build every output that would be affected by your change and no more. Incorrect dependencies can lead to building too much (wasting time) or worse, building too little (wasting way more time). Unfortunately keeping your dependencies specified in your build configuration is time consuming and error-prone.

Tom Tromey invented a technique called auto-dependency generation whereby GCC can output a Makefile formatted file when it compiles sources that describes the inputs and outputs from that step. GNU Make will include these generated files, if they exist. Idea has been adopted by other compilers and other build systems.

Unfortunately not all tools can describe what they do in terms of Makefile rules – they’re too busy doing whatever they’re supposed to do. For a long time I’ve wanted to build something that would observe file I/O and generate dependency rule files based on what was observed. I read a bunch of the GNU Make and Ninja source code, I researched loopback filesystems, I learned about various system-level profiling technologies.

Then the other day I wrote a short shell script: strace-deps

It uses strace to trace the system calls that a process (and its children) makes, then looks at all the openat invocations (since that appears to be the syscall that a modern libc uses) to find the files that were written and the files that were read. It excludes uninteresting files (/tmp/, /dev/, /var/, /usr/, etc) and then generates a dependency file describing what happened.

It needs to be explicitly invoked and the dependency file needs to be explicitly specified. My rule for iverilog compilation of Verilog testbenches looks like:
%_tb: %_tb.v
strace-deps $@.d iverilog -o $@ $<
So for a testbench foo_tb.v when generating foo_tb with iverilog it will observe all the file I/O and generate foo_tb.d based on what strace-deps sees.

This isn’t perfect but it’s definitely improved my development experience.