[Geany-Devel] Helping Geany move forward: testing

[Lex Trotman]

Hi Vasily,

On 28 April 2017 at 06:51, Vasiliy Faronov <vfaronov at gmail.com> wrote:
> Hi all,
>
> From discussions elsewhere, such as [1], it sounds like one of the
> things holding back Geany development right now is a need for more
> testing.

I can only speak from my point of view, but I believe it probably
applies to at least some other committers.  I can afford a few minutes
at a time through the day to answer emails and IRC and comment on PRs.
Even inspect the code for simple PRs (which I mark LGBI Looks Good By
Inspection).

But I can't often afford a bigger block of time to confirm the fault
exists in my system, grab the PR, test the fault no longer exists,
merge to master, ooops, master is not up to date, pull changes, back
to merge, push to github.  Sure that process is more polished for
those who have the time to commit more often, but they are also short
of time.

Personally I am willing to skip testing myself for PRs that are
complete and immediately committable, not too complex, or
controversial and its been tested by someone who seems reliable (other
than the OP, its way too easy to miss problems on your own work).

Also I can (have to) accept others testing when I don't have the setup
to test (Windows, weird desktops, and networked file systems being the
prime examples).

This is the approach I use for another project where I don't want to
load the Gb of dependencies to fully test changes on this machine.

>
> I have some spare time that I can dedicate to exploratory testing of
> PRs to Geany and Geany-Plugins. I'm not a QA professional, but I am a
> programmer, I use a range of Geany features daily, and I understand
> Geany's code.

Testing when you use it is by far the best way, so long as your usage
includes the feature (one of the issues with the PR you referenced,
its off by default and simply not wanted in my workflow, so it will
only ever get artificial testing).

>
> How can I test PRs in a way that would really help them get merged?
>
> In particular:
>
> 1. How can I determine that a PR is mostly blocked on testing, and is
> likely to be merged when positive testing results come in? Some PRs
> are marked as "approved" in GitHub yet are not merged -- is that it?

The review and approval system in github is pretty new and we don't
use it consistently yet, so no its not a reliable indicator.
Unfortunately its probably going to require slightly more judgement:

1. has someone checked it and posted that it looks ok (I try to use
LGBI consistently but its not universal)

2. have any requested changes been made, you will find there are a
distressingly high proportion of PRs where a small change by the OP
would make it committable, but they don't seem to do it.

3. is it non-controversial, or has it come to a consensus (like the
one you referenced)

>
> 2. How can I communicate my results to the satisfaction of Geany
> committers? For example, I could write up some kind of a report: an
> outline of what I tried, with screenshots of what I got -- would that
> help?

For simple PRs just posting "I have been running with this for the
last week/month/whatever using it often and it works fine" is likely
to be sufficient.  For more complex ones, then a description of how
you tested it is likely to be needed.  Screenshots would only be
relevant if the purpose of the change was to affect the way something
looked.

Things that interact with the operating system or files are the most
difficult since they should also be tested on Windows (which the Geany
team don't regularly use) and/or remote filesystems (there seem to be
a lot of users of SSHFS in particular).

In all cases just posting on the PR "I am testing this will report
back" is good.  Then if anything special is needed or if its not ready
somebody will probably notice and post a request.

Thanks
Lex

>
> Thank you.
>
>
> [1] https://github.com/geany/geany/pull/1246#issuecomment-290047712
>
>
> --
> Vasiliy
> _______________________________________________
> Devel mailing list
> Devel at lists.geany.org
> https://lists.geany.org/cgi-bin/mailman/listinfo/devel