[Github-comments] [geany-plugins] Automake (#277)

Colomban Wendling notifications at xxxxx
Tue Dec 1 17:11:10 UTC 2015


Yeah unfortunately Automake doesn't (yet) have a way to ask `SUBDIRS` to be built in parallel.  It would indeed work for GP's top-level Makefile, but unfortunately there's no easy way to do that.
Non-recursiveness is also (slightly) more efficient than recursiveness just because it wastes less time spawning processes, analyzing files mdates and reading Makefiles, but that's probably less of a concern -- although on my machine (no SSD) running `make` on GP when there's nothing to do still takes around 3.5s (parallelized or not).

All this said, when I was writing the initial comment on SUBDIRS not being parallelizable, I had an idea for a hack to work around this.  It's not really beautiful, but it does the job:
```diff
$ git diff
diff --git a/Makefile.am b/Makefile.am
index 8cf1634..148c67e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -176,3 +176,10 @@ EXTRA_DIST = \
        wscript \
        README.waf \
        README.windows
+
+
+all-recursive: $(SUBDIRS)
+.PHONY: $(SUBDIRS)
+
+$(SUBDIRS):
+       $(am__cd) $@ && $(MAKE) $(AM_MAKEFLAGS) all
```

What it does is recursing in each `$(SUBDIRS)` first, before making all-recursive.  This obviously isn't prefect, as it is effectively slower when there's nothing to do, as it'll recurse in each `$(SUBDIRS)` twice; but it will parallelize the first run (as per `-j` option), so the second one won't do anything, meaning the actual build will be parallelized.
On my machine (2 physical, 4 logical cores), such an empty run takes around 7s when unparallelizd, and less than 5s when parallelized (`-j4`).
OTOH, a full build with `-j4` goes from around 68s to around 53s (-15s, 22%).  FTR, with CCache (so the actual build time is a LOT smaller), it goes from around 24s to around 16s (-8s, 33%).

This can also be done in a slightly more hackish way outside of Automake's Makefile, meaning it doesn't change how an Automake's Makefile run behaves.  Using another Makefile (I named it *makefile.parallel* locally), like that:
```make
#!/usr/bin/make -f
#
# Parallelizes the top-level Makefile of a SUBDIR-based Automake build system.
#
# What it actually does is recursing in $(builddir) for each target, and in
# each $(SUBDIRS) for target 'all'.  'all' has a dependency on $(SUBDIRS) so
# they are recursed in first, respecting `-j` option as any target.
#
# Note that this means that each $(SUBDIRS) is first built in parallel, and
# then again through the top-level Makefile for target 'all'.  This is normally
# not so much of a problem, as the targets should be up-to-date the second
# time, so you only get the Make overhead but no actual building.
# This is needed to "recurse" in '.' in case it's useful.
#
# As this makes uses of normal Make features, it is very possible to force some
# ordering by declaring explicit dependencies between some elements of
# $(SUBDIRS); for example if sub-directory 'po' depends on recursing in another
# one first, i.e. 'src', declare '$(builddir)/po: $(builddir)/src' (not
# forgetting '$(builddir)/' prefix), so 'src' is build before 'po'.
#
# Only 'all' is parallelized.  First, this is because it's likely to be the
# only target that really makes sense to parallelize, as it's the one building
# real things.  Also, as the parallelized targets are run twice (see above),
# a second run should have low overhead, so it doesn't make sense to e.g.
# parallelize i.e.g 'check' the same way, as it would run the checks twice --
# although parallelization would slightly help when building test programs.
# This could be addressed by omitting recursion in '.', but this can be a
# problem if './Makefile' contains anything beside recursion.
# Additionally, there's a technical difficulty in knowing which target was
# actually asked for, as each $(SUBDIRS) is a target itself, so $@ can't be
# relied on.
#
# Limitations:
# * Inspecting the value of Automake's SUBDIRS variable is naive, and doesn't
#   support expansions and the like.  This means that if $(SUBDIRS) from
#   Makefile.am contains variable references it'll most likely will break.

# user can set either builddir (when running from the source directory) or
# srcdir (when running from the build directory).  If builddir==srcdir, no
# overrides are needed.
builddir ?= .
srcdir ?= .

upper_level = $(srcdir)/Makefile.am

SED ?= sed

# extract SUBDIRS
SUBDIRS != $(SED) -ne '/SUBDIRS/s%^.*= \(.*\$\)%$(builddir)/\1%p' $(upper_level)

# MAKEFLAGS minus builddir= and srcdir=
filtered_makeflags = `echo $(MAKEFLAGS) | \
                      $(SED) -e 's/\(build\|src\)dir=[^ ]*//g'`

# all targets we want to forward somehow
TARGETS := all check clean distclean dist distcheck

all: $(SUBDIRS)

$(TARGETS):
	$(MAKE) -C $(builddir) $(filtered_makeflags) $@

$(SUBDIRS):
	$(MAKE) -C $@ $(filtered_makeflags)

.PHONY: $(SUBDIRS) $(TARGETS)
```

---
Reply to this email directly or view it on GitHub:
https://github.com/geany/geany-plugins/pull/277#issuecomment-161035919
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.geany.org/pipermail/github-comments/attachments/20151201/292367a1/attachment.html>


More information about the Github-comments mailing list