[Geany-Devel] Helping Geany move forward: testing

[Matthew Brush]

On 2017-04-28 02:35 PM, Thomas Martitz wrote:
> Am 27.04.2017 um 22:51 schrieb Vasiliy Faronov:
>> Hi all,
>>
>>  From discussions elsewhere, such as [1], it sounds like one of the
>> things holding back Geany development right now is a need for more
>> testing.
>
> Helping to test PRs is truly needed, and much appreciated.
>
> However, I do think that Geany lacks also actual developers that cna
> merge stuff. I feel the current team is afraid of merging non-trivial
> changes, leaving even semi-complex patches to Colomban. Unfortunately
> Colomban has little time these days, too, so we're kind of stuck. There
> are lots of PRs that have recent activity from the authors and are
> tested appropriately but still don't get attention from developers.
>

My general problem is that we don't have a unstable/development branch 
per se, nor proper automated testing, and I don't want to break master 
so I won't merge a single thing without testing it thoroughly myself. 
This can turn a 5-10 minute merge into a several hours or more testing 
session, requiring special setups and re-compiling Geany on 3 different 
OSes, etc.

Travis CI is great, but unless it can run make check with loads of 
static analysis and runtime analysis while it runs unit tests and such, 
it's basically just saying the code compiles. As we all know, it's 
relatively easy to make C code that compiles but crashes horribly at 
runtime in weird corner cases (off by one, null deref, etc.).

Personally I'd feel a lot better merging PRs I haven't thoroughly tested 
if we had:

   - Clang static analyzer during the build
   - A Git hook or manual use of clang-format or other formatter to 
prevent the "extra white space" or "wrong comment style" type of issues 
that commonly occur in PRs.
   - Ability for PRs to come with tests (requires testing support).
   - Linking in Clang's address & memory sanitizers while running all of 
the tests.

Just some thoughts.

Regards,
Matthew Brush