Hi, I'm working on updating the Geany package in nixpkgs from 1.38 to 2.0: https://github.com/NixOS/nixpkgs/pull/265362.
With Geany 2.0, [`tests/test_sidebar.c`](https://github.com/geany/geany/blob/2.0.0/tests/test_sidebar.c) fails on nixpkgs' automatic PR builder (ofborg) on macOS, but not on macOS locally - presumably due to the environment being headless in the former case. On Linux, `test_sidebar` succeeds both locally and on the builder. The issue would likely show up on Hydra as well, our central CI service, and would cause us to introduce workarounds that are probably avoidable.
The return value of [`gtk_init_check` here](https://github.com/geany/geany/blob/2.0.0/tests/test_sidebar.c#L145) is never examined. https://docs.gtk.org/gtk3/func.init_check.html says:
_"Note that calling any GTK function or instantiating any GTK type after this function returns FALSE results in undefined behavior."_
Could you please take a look at this? I can't really gauge what it would take to make the test run properly on Darwin in a headless environment (if possible). Worst case the test could skip itself and emit an appropriate message if GTK can't initialize, which would still be better than downstream having to explicitly work around it.
---
The build log for `x86_64-darwin` with the failing test can be viewed [here](https://logs.ofborg.org/?key=nixos/nixpkgs.265362&attempt_id=2ac562f8-63...). I attached it here in case that link becomes inaccessible: [geany_2.0_x86_64-darwin_ofborg_buildlog.txt](https://github.com/geany/geany/files/13258067/geany_2.0_x86_64-darwin_ofborg...)
Well, the test program should test the result of `gtk_init_check()` and early out, but what to return, if it returns failure any environment where it doesn't init is left with a failure to handle, but on the other hand return success isn't right either as it gives the impression the test ran. I can see why @kugel- ignored it.
My personal approach would be to return fail unless the whole test ran, and environments where the test cannot run need to disable it.
I was thinking it could work with [`g_test_skip`](https://docs.gtk.org/glib/func.test_skip.html) (_"Indicates that a test was skipped."_), like [used here](https://github.com/geany/geany/blob/2.0.0/tests/test_sidebar.c#L71). It looks like that isn't a GTK function, thus it should be safe to call.
IIUC the call to `gtk_init_func()` is not inside a test, so `g_test_skip()` will do nothing. The example you pointed out is inside a function that is registered as a test. And that still leaves the question of what to return to your test environment, pass or fail. As I said, returning pass when none of the test ran isn't useful.
You can return `77` to tell Automake that the test was skipped.[^1][^2]
[^1]: https://docs.gtk.org/glib/func.test_run.html [^2]: https://www.gnu.org/software/automake/manual/automake.html#index-Exit-status...
I guess if "somebody" made a PR it would get accepted.
On headless systems you should run the tests under Xephyr or something similar. That would be preferable over skipping the tests.
That's what I do when I build geany on my local Jenkins instance.
@kugel- Bases on e.g. [here](https://wiki.archlinux.org/title/Xephyr): _"Xephyr is a nested X server that runs as an X application."_ So using that would require an existing X server in the first place?
At any rate, I think such an approach can be a good solution and it's good that you do this, it's just overkill to add downstram just for that one test.
I proposed an implementation of what I had in mind [here](https://github.com/geany/geany/pull/3676) and hope it could be considered :) For the time being, I am probably going to explicitly skip the test in nixpkgs.
Sorry, I mixed it up. I mean [Xvfb](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml). That's a pure software emulated X environment.
Why do you execute the tests at all?
Xvfb
Thanks.
Why do you execute the tests at all?
Basically, I saw no reason to disable anything inside Geany's test suite until now, or disabling running the test suite altogether (but then again, it's only since a couple months that I'm active to some degree in nixpkgs). It was enabled since Geany's [initial addition](https://github.com/NixOS/nixpkgs/commit/51fe78841797c3d59ed7289df28ed19715c9...) to nixpkgs and seems to have been fine since.[^1][^2] The rest is just me liking to get to the bottom of things sometimes. ^^;
I think it doesn't hurt to run Geany's suite in nixpkgs as it's relatively quick and lightweight—as long as it doesn't break things and cause additional maintenance burden.
[^1]: Some context: running upstream test suites is by default disabled in nixpkgs; package(r)s have to explicitly opt into this, ordinarily with `doCheck = true;`. There was come discussion around enabling this by default, as there are arguably good reasons for both approaches (notably https://github.com/NixOS/nixpkgs/issues/33599 and https://github.com/NixOS/rfcs/pull/95), but it seems to have petered out. [^2]: But "we've always been doing it this way" is never a good argument.
I still recommend Xvfb. Geany is a GTK program, so you can expect that future tests will also require successful GTK initialization.
But a PR that skips the test would probably be accepted. Do you know if the `exit 77` trick works on meson as well?
Do you know if the exit 77 trick works on meson as well?
It should work: https://mesonbuild.com/Unit-tests.html#skipped-tests-and-hard-errors
But it's not a "trick", it's the GNU standard approach (as mentioned there).
I still recommend Xvfb. Geany is a GTK program, so you can expect that future tests will also require successful GTK initialization.
Yeah, thanks. I will consider it. Other builds in nixpkgs are already using it in their `checkPhase`.
@kugel-: the approach works with Geany's Meson-based build too: https://github.com/geany/geany/pull/3676#issuecomment-1796529112
github-comments@lists.geany.org