I know this isn't a Geany issue directly, but maybe there's something that can be done to improve the build times. I ran some tests to measure the actual build times.
## Machines
* "Linux Native" * Ubuntu 18.04 * 32GB RAM * 6-core i7 @ 3.7GHz * Entirely on SSD * "Windows with MSYS2" * Windows 10 Pro * 32GB RAM * 6-core i7 @ 3.7GHz * Geany source code on SSD, system headers and build tools on mechanical HDD. * "Linux VM Guest on Windows Host" * ElementaryOS from 2018, in VirtualBox on Windows 10 * 4GB RAM * 1-core emulated CPU (presumably 3.7GHz + VM overhead) * All files in virtual disk stored on a mechanical HDD.
## Tests
* "configure" - `./configure [opts]` * "make" - `make -j12` after a `make clean` * "imake" - `touch src/build.h && make -j12` to do an incremental build * "install" - `make -j12 install`
I used 12 `make` jobs on account of the 12 hyperthreads in the CPU, even in the VM which makes no sense since I only assigned it a single core. I used `time` on the above commands and took the average of 3 runs for each. All 3 runs were quite similar in all cases.
## The results
The Y axis is in seconds.
## Observations
* As expected `configure` was really slow on Windows since it spawns tons of processes. * The Windows build seems to be spending a large portion of the time linking which is why even incremental build was still very slow. * In the wimpy Linux VM with 1 core, the vast majority of the compile time was spent compiling Scintilla's C++ code, shown by how the incremental build that didn't touch C++ code is comparatively fast. * I don't know whether it's the `ld` linker which is slow, the `libtool` stuff, or some combination.
## Possible Solutions?
The linking times even in the Libtool helper libraries is really slow, so maybe for Windows we could not use helper libraries and link all of the `libgeany` objects together in one go.
Maybe there's a way to get linking itself to be faster, needs investigation. Maybe a different build of `libtool` or something?
Agree its known that autotools configure is slow on windows because of the process spawning, but its only run once in a blue moon so I doubt there is much return in improving it.
One comment on the VM case, its a known issue (benchmarks on Phoronix and acknowledged by MS for WSL) that Linux on Windows file IO is slow (compared to native Linux) and could account for much of the slowness there. That also agrees with linking being slow, since it is the one process that accesses the most files, all the `.o` ones. Also there is the thing of creating `.o` and `.lo` files for the same source.
Not sure why the incremental build is slower on windows native, are you sure its building the same things? And of course GNU ld linking to windows libraries via MSYS2 might not be terribly optimised??
The graph is misleading, it puts runs with differing core counts on the same scale, can't you run your VM with the same number of cores as the rest? Or run everything with `-j1` so its all comparable.
I can't really believe I could be suggesting something like this, but anyone tried [meson](https://mesonbuild.com/) or such? I'm not totally opposed against moving away from Autotools in the unlikely event there's a decent alternative (e.g. that does everything we need from AT [1], and has actual advantages), although every time I saw a new build system claiming this it fell short in one way or the other. Just an idea, because I unfortunately doubt we can make libtool fast on Windows, and we need at least libgeany being a shared library for plugins. Or maybe things like ninja would help a little? (I doubt it, as it should still run the same commands)
[1] maybe the most common shortcomings are distribution tarballs and painless portability to exotic Unices.
Speaking as someone who uses and likes meson because its a _developer_ build tool, (AT isn't it says so right on its docs, its a distribution tool). Unfortunately any alternative probably needs a "try it and see" approach to finding if its faster than AT, because its likely to only be a speed improvement for something Geanys size if it runs fewer commands (which it may do for the library stuff). I don't think, but it needs to be proved, that there will be much speed improvement due to filesystem metadata access that `make` does that `ninja` doesn't for Geany (but its very important for say chromium).
@b4n:
distribution tarballs
`ninja dist` according to the meson site
As for exotics, well, if they want Geany they can contribute a build system, its all open source.
IIRC somebody did offer to make a meson port in the past, but I can't remember who (and my googling didn't find it either, @codebrainz google services might :).
I ported Geany to meson a few months back. I think it wasn't complete (docs etc. missing) but geany and libgeany.so builds and runs fine.
I'm not a fan of meson, though. But I can live with it.
@kugel- I don't suppose you happened to try your meson on windows and can compare the speed?
I guess the first thing to do is see if we can figure out if it's Libtool being slow or just GNU Linker.
@ntrel is using a plain Make file which is presumably using GNU LD without Libtool, so maybe could comment on whether the linking stage is appreciably faster. If it is just a Libtool issue, I think we could probably just not use it (at least on Windows)?
The graph is misleading, it puts runs with differing core counts on the same scale, can't you run your VM with the same number of cores as the rest? Or run everything with -j1 so its all comparable.
No, it was painfully slow and by the time I realized I was half way through the test. The only reason for the VM was just to show how a slow Linux VM on a slow disk could outperform the Windows build for most cases (I didn't expect it would take so long compiling Scintilla's C++ code).
@ntrel is using a plain Make file which is presumably using GNU LD without Libtool, so maybe could comment on whether the linking stage is appreciably faster
My makefiles are much faster at `make foo.o`, very roughly ten times faster. That's my main problem.
Btw I'm using -O0 throughout on both.
How does linking compare between the Autotools Makefile and your own? On my computer compilation was nearly as fast as on my Linux partition, but linking took ages.
I updated the test/graph to include Windows Subsystem for Linux, which is actually really cool.
I ported Geany to meson a few months back. I think it wasn't complete (docs etc. missing)
If you provide the `meson.build` file(s), I can try to add it to the tests to see how it compares. I disabled docs and stuff for the tests anyway.
I did start tinkering with a Meson build myself but I didn't go far since Autotools works fine for me.
I would expect if Meson outputs a build system using native MS proprietary but free$ toolchain, that it would beat MSYS2 and WSL by a wide margin.
My makefiles are much faster at make foo.o, very roughly ten times faster.
If you provide them I will add to tests for comparison. I don't have any since the libgeany stuff.
Linking libgeany and linking geany are pretty slow with my makefiles too.
https://github.com/ntrel/geany/commits/makewin32
Unfortunately WSL needs Windows 10, but good to know it builds fast.
Also I will just throw an observation out there that trying to minimise headers may not be the way to go.
The C++ project I'm working on compiled significantly faster when it moved from individual includes to a single include which included _all_ the others. At the time I didn't measure the improvement or investigate why that happened. I was just happy as I was actually expecting it to be slightly slower.
Perhaps GCC or meson uses some sort of caching which it can activate if it observes the same includes, which the single include guarantees, and guarantees to be in the same order.
On the other hand if the linking is the slow part twiddling the includes won't help. @ntrel can you put figures on your "slow" so we know its not just that you are nearly as impatient as me.
I updated the Linux VM test with proper specs to make it more comparable.
Updated to show results of @ntrel's Make files
Precompiled headers could definitely help. I once saw a big improvement from precompiling just the GTK headers. @elextr perhaps that happened in your scenario?
@kugel- yeah, thats what I suspect, but as I said I havn't actually investigated closely, although I thought it took some effort and special options to make `.pch` files.
Also `make -j` and `CC=clang CXX=clang++` _really_ speeds Geany compile up on Linux.
[Warning: `make -j` spawns many processes (I counted 50+ and couldn't scroll `top` fast enough to get them all) so you need memory and SSD]
@elextr
if the linking is the slow part twiddling the includes won't help
As I said, my main problem with autotools is the huge time taken to do `make build.o` (or whatever file I'm mainly working on). I almost never rebuild scintilla. Speeding linking would be very useful, but that's less important for me.
@ntrel can you put figures on your "slow"
Thanks to Matthew's updated graph, you can see that his imake in yellow is *fifteen times* slower with autotools vs my makefiles.
Reducing header includes doesn't help a lot, but worth doing as it's so easy and has no downside.
imake in yellow is fifteen times slower with autotools vs my makefiles.
Even so its less than 3 minutes. Getting up and stretching is good for you :)
Reducing header includes doesn't help a lot, but worth doing as it's so easy and has no downside.
@b4n noted the downsides when rejecting #2299.
Frankly everybody else develops on Linux where speed is good, so they have no need for machinations to improve compilation speed a few seconds. Perhaps you should investigate developing on Linux and only building on Windows when its working, or if you don't want to got all the way develop using one of the options @codebrainz identified for Linux on windows that have better performance than pure windows.
Or give yourself an early birthday present of a faster machine :smile:
Even so its less than 3 minutes.
For one object file maybe.
I already said WSL requires Windows 10. I have 7. Do the other options require a massive download?
Why would I bother to install a new OS when my makefiles are fast enough?
give yourself an early birthday present of a faster machine
Since September I work ten hours a week for the London living wage. Before that I hadn't been employed since 2005. If I really wanted I could get a newer machine but why should I, the hardware is sufficient to work on geany. New electronics have a big carbon cost.
Even so its less than 3 minutes.
For one object file maybe (with autotools. exaggerating slightly).
Just to be clear, you are saying it takes three minutes to compile just `build.c` using your makefile?
So its a communication problem, this is the first time you have actually provided a number to explain why you want to make the change, up to now all we have seen is @ntrel wants to make changes to speed up compile for his own unexplained purposes. Changes that are imposed on everyone, and, as @b4n explained, are considered poor design and poor for long term maintainability, and which nobody else seems to need.
To be clear, people are trying to help, suggesting things that have worked for them, and @codebrainz has spent considerable effort to analyse many options, but without numeric information about your configuration we were just shooting in the dark.
Why would I bother to install a new OS when my makefiles are fast enough?
1. I presume you mean "fast enough except that compiling any one file takes way too long :grin:" which is a function of the system rather than the build tool, and
2. Since Windows 7 extended support is EOL on January 2020, ie 3 months time, when security and other support ceases. You probably need to consider alternatives for more reasons than compiling Geany. I don't know what other software you use and if its practical for you to move the computer to Linux, but its likely better on low performance hardware than Windows 10 would be, and is free if you can find a free wifi (library, cafe) to download it.
New electronics have a big carbon cost.
Well, I certainly don't want to make you responsible for killing Johnny Polar Bear :cry: although data on relative environmental costs of building new hardware vs the much lower base power use of new hardware and the lower time it runs at full power to compile Geany would be interesting, but not a topic for this forum.
I'm not going to post the results since it takes a long time to run and had no advantage (actually slightly worse on Linux); but just as an experiment I flattened the build into a single Makefile.am and listed all sources for libgeany/geany directly so it only built and linked the libgeany library and the geany binary without intermediate helper libraries or recursing into the handful of subdirectories.
At this point I'm starting to think the Autotools build in MSYS2 just isn't making use of my system fully, it feels as slow as a serial build on Linux. Even with `make -j` it spawns hundreds of processes and prints the messages fast, and then pegs the CPU while compiling everything (with no printing), and then when it prints the message it's linking libgeany, the CPU goes almost idle and not much disk IO, as if it's just stuck there.
I wonder if running the libtool command that does the linking manually shows it as well or if it's a make-related problem.
Is there any way not to run libtool individually on each object file?
@elextr
I presume you mean "fast enough except that compiling any one file takes way too long
No, that's with autotools. Yeah I could put Linux on it sometime, but it's a hassle.
No, that's with autotools.
Autotools makes the actual compile slow? Or is it the preparation for the compile?
Here's meson branch. It doens't quite build because I needed fox merge conflicts fpr .135
The branch does not compile, haps someone beats me to it.
https://github.com/kugel-/geany/tree/meson
(it compiled in the past but it's too late for me to make the adoption for latest geany master.
@codebrainz the branch builds now (for me anyway; it even runs). Give it a try.
@kugel- seems to work OK on Linux, but doesn't work on MSYS2, or if it's meant for use without MSYS2, then how would I setup such an environment?
FWIW, [here's a Gist](https://gist.github.com/codebrainz/02f50254cd527a6a00b6bc1593378851) of some random changes I had to make, but it still won't link due to undefined references, probably because of the visibility stuff, which didn't work before the change either.
Could you change the symbol visibility back to default temporarily?
You mean to comment out all the `GEANY_API_SYMBOL` stuff? I can try next time I boot into Windows I guess, but I did mess around with it for quite a while trying to get it to work. I get really frustrated and annoyed trying to develop on Windows, especially when it's on stuff I don't know.
No, I mean using the -fvisibility=hidden option
*not using*
Windows sucks, WSL and even a VM had better results. @codebrainz you can close this.
github-comments@lists.geany.org