<p>Yeah unfortunately Automake doesn't (yet) have a way to ask <code>SUBDIRS</code> to be built in parallel. It would indeed work for GP's top-level Makefile, but unfortunately there's no easy way to do that.<br>
Non-recursiveness is also (slightly) more efficient than recursiveness just because it wastes less time spawning processes, analyzing files mdates and reading Makefiles, but that's probably less of a concern -- although on my machine (no SSD) running <code>make</code> on GP when there's nothing to do still takes around 3.5s (parallelized or not).</p>
<p>All this said, when I was writing the initial comment on SUBDIRS not being parallelizable, I had an idea for a hack to work around this. It's not really beautiful, but it does the job:</p>
<div class="highlight highlight-source-diff"><pre>$ git diff
<span class="pl-c1">diff --git a/Makefile.am b/Makefile.am</span>
index 8cf1634..148c67e 100644
<span class="pl-md">--- a/Makefile.am</span>
<span class="pl-mi1">+++ b/Makefile.am</span>
<span class="pl-mdr">@@ -176,3 +176,10 @@</span> EXTRA_DIST = \
wscript \
README.waf \
README.windows
<span class="pl-mi1">+</span>
<span class="pl-mi1">+</span>
<span class="pl-mi1">+all-recursive: $(SUBDIRS)</span>
<span class="pl-mi1">+.PHONY: $(SUBDIRS)</span>
<span class="pl-mi1">+</span>
<span class="pl-mi1">+$(SUBDIRS):</span>
<span class="pl-mi1">+ $(am__cd) $@ && $(MAKE) $(AM_MAKEFLAGS) all</span></pre></div>
<p>What it does is recursing in each <code>$(SUBDIRS)</code> first, before making all-recursive. This obviously isn't prefect, as it is effectively slower when there's nothing to do, as it'll recurse in each <code>$(SUBDIRS)</code> twice; but it will parallelize the first run (as per <code>-j</code> option), so the second one won't do anything, meaning the actual build will be parallelized.<br>
On my machine (2 physical, 4 logical cores), such an empty run takes around 7s when unparallelizd, and less than 5s when parallelized (<code>-j4</code>).<br>
OTOH, a full build with <code>-j4</code> goes from around 68s to around 53s (-15s, 22%). FTR, with CCache (so the actual build time is a LOT smaller), it goes from around 24s to around 16s (-8s, 33%).</p>
<p>This can also be done in a slightly more hackish way outside of Automake's Makefile, meaning it doesn't change how an Automake's Makefile run behaves. Using another Makefile (I named it <em>makefile.parallel</em> locally), like that:</p>
<div class="highlight highlight-source-makefile"><pre><span class="pl-c">#!/usr/bin/make -f</span>
<span class="pl-c">#</span>
<span class="pl-c"># Parallelizes the top-level Makefile of a SUBDIR-based Automake build system.</span>
<span class="pl-c">#</span>
<span class="pl-c"># What it actually does is recursing in $(builddir) for each target, and in</span>
<span class="pl-c"># each $(SUBDIRS) for target 'all'. 'all' has a dependency on $(SUBDIRS) so</span>
<span class="pl-c"># they are recursed in first, respecting `-j` option as any target.</span>
<span class="pl-c">#</span>
<span class="pl-c"># Note that this means that each $(SUBDIRS) is first built in parallel, and</span>
<span class="pl-c"># then again through the top-level Makefile for target 'all'. This is normally</span>
<span class="pl-c"># not so much of a problem, as the targets should be up-to-date the second</span>
<span class="pl-c"># time, so you only get the Make overhead but no actual building.</span>
<span class="pl-c"># This is needed to "recurse" in '.' in case it's useful.</span>
<span class="pl-c">#</span>
<span class="pl-c"># As this makes uses of normal Make features, it is very possible to force some</span>
<span class="pl-c"># ordering by declaring explicit dependencies between some elements of</span>
<span class="pl-c"># $(SUBDIRS); for example if sub-directory 'po' depends on recursing in another</span>
<span class="pl-c"># one first, i.e. 'src', declare '$(builddir)/po: $(builddir)/src' (not</span>
<span class="pl-c"># forgetting '$(builddir)/' prefix), so 'src' is build before 'po'.</span>
<span class="pl-c">#</span>
<span class="pl-c"># Only 'all' is parallelized. First, this is because it's likely to be the</span>
<span class="pl-c"># only target that really makes sense to parallelize, as it's the one building</span>
<span class="pl-c"># real things. Also, as the parallelized targets are run twice (see above),</span>
<span class="pl-c"># a second run should have low overhead, so it doesn't make sense to e.g.</span>
<span class="pl-c"># parallelize i.e.g 'check' the same way, as it would run the checks twice --</span>
<span class="pl-c"># although parallelization would slightly help when building test programs.</span>
<span class="pl-c"># This could be addressed by omitting recursion in '.', but this can be a</span>
<span class="pl-c"># problem if './Makefile' contains anything beside recursion.</span>
<span class="pl-c"># Additionally, there's a technical difficulty in knowing which target was</span>
<span class="pl-c"># actually asked for, as each $(SUBDIRS) is a target itself, so $@ can't be</span>
<span class="pl-c"># relied on.</span>
<span class="pl-c">#</span>
<span class="pl-c"># Limitations:</span>
<span class="pl-c"># * Inspecting the value of Automake's SUBDIRS variable is naive, and doesn't</span>
<span class="pl-c"># support expansions and the like. This means that if $(SUBDIRS) from</span>
<span class="pl-c"># Makefile.am contains variable references it'll most likely will break.</span>
<span class="pl-c"># user can set either builddir (when running from the source directory) or</span>
<span class="pl-c"># srcdir (when running from the build directory). If builddir==srcdir, no</span>
<span class="pl-c"># overrides are needed.</span>
<span class="pl-smi">builddir</span> ?= .
<span class="pl-smi">srcdir</span> ?= .
<span class="pl-smi">upper_level</span> = <span class="pl-s">$(<span class="pl-smi">srcdir</span>)</span>/Makefile.am
<span class="pl-smi">SED</span> ?= sed
<span class="pl-c"># extract SUBDIRS</span>
SUBDIRS != $(SED) -ne '/SUBDIRS/s%^.*= \(.*\$\)%$(builddir)/\1%p' $(upper_level)
<span class="pl-c"># MAKEFLAGS minus builddir= and srcdir=</span>
<span class="pl-smi">filtered_makeflags</span> = <span class="pl-s"><span class="pl-pds">`</span><span class="pl-s1"><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">$(</span>MAKEFLAGS<span class="pl-pds">)</span></span> <span class="pl-k">|</span> \</span></span>
<span class="pl-s"><span class="pl-s1"> <span class="pl-s"><span class="pl-pds">$(</span>SED<span class="pl-pds">)</span></span> -e <span class="pl-s"><span class="pl-pds">'</span>s/\(build\|src\)dir=[^ ]*//g<span class="pl-pds">'</span></span></span><span class="pl-pds"><span class="pl-s1">`</span></span></span>
<span class="pl-c"># all targets we want to forward somehow</span>
<span class="pl-smi">TARGETS</span> := all check clean distclean dist distcheck
<span class="pl-en">all</span>: <span class="pl-s">$(<span class="pl-smi">SUBDIRS</span>)</span>
<span class="pl-en"><span class="pl-s">$(<span class="pl-smi">TARGETS</span>)</span></span>:
$(MAKE) -C $(builddir) $(filtered_makeflags) $@
<span class="pl-en"><span class="pl-s">$(<span class="pl-smi">SUBDIRS</span>)</span></span>:
$(MAKE) -C $@ $(filtered_makeflags)
<span class="pl-c1">.PHONY</span>: <span class="pl-s">$(<span class="pl-smi">SUBDIRS</span>)</span> <span class="pl-s">$(<span class="pl-smi">TARGETS</span>)</span></pre></div>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/geany/geany-plugins/pull/277#issuecomment-161035919">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/ABDrJ3UP4VcekhylPzFoQqBiyklsx6IKks5pLcwugaJpZM4Fg-b2.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/geany/geany-plugins/pull/277#issuecomment-161035919"></link>
<meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>