[Geany-Devel] Helping Geany move forward: testing

[Matthew Brush]

On 2017-04-28 05:35 PM, Lex Trotman wrote:
> On 29 April 2017 at 09:55, Matthew Brush <mbrush at codebrainz.ca> wrote:
>> On 2017-04-28 02:35 PM, Thomas Martitz wrote:
>>>
>>> Am 27.04.2017 um 22:51 schrieb Vasiliy Faronov:
>>>>
>>>> Hi all,
>>>>
>>>>  From discussions elsewhere, such as [1], it sounds like one of the
>>>> things holding back Geany development right now is a need for more
>>>> testing.
>>>
>>>
>>> Helping to test PRs is truly needed, and much appreciated.
>>>
>>> However, I do think that Geany lacks also actual developers that cna
>>> merge stuff. I feel the current team is afraid of merging non-trivial
>>> changes, leaving even semi-complex patches to Colomban. Unfortunately
>>> Colomban has little time these days, too, so we're kind of stuck. There
>>> are lots of PRs that have recent activity from the authors and are
>>> tested appropriately but still don't get attention from developers.
>>>
>>
>> My general problem is that we don't have a unstable/development branch per
>> se, nor proper automated testing, and I don't want to break master so I
>> won't merge a single thing without testing it thoroughly myself. This can
>> turn a 5-10 minute merge into a several hours or more testing session,
>> requiring special setups and re-compiling Geany on 3 different OSes, etc.
>
> I have to agree with Matthew that:
>
> 1. Nobody wants to break master because its what everybody is using.
> Problem is that if we had a development branch nobody would be using
> it because it might break, so its insufficiently tested.  I don't have
> a solution to that.
>
> 2. I am more willing to accept others testing and to make a judgement
> call about testing on all platforms.  I have used that approach
> successfully on other projects where I couldn't personally test some
> configurations.  But I understand where Matthew is coming from
> regarding the amount of work to do a good testing job.
>
> 3. A thorough test is becoming too big a job, and that is even worse
> for the more complex PRs that Thomas mentions.  Simply don't have the
> time.  And for changes to the plugin interface that need a plugin to
> test, well, unless the OP provides such a plugin, it just isn't going
> to happen.
>
>
>> Travis CI is great, but unless it can run make check with loads of static
>> analysis and runtime analysis while it runs unit tests and such, it's
>> basically just saying the code compiles. As we all know, it's relatively
>> easy to make C code that compiles but crashes horribly at runtime in weird
>> corner cases (off by one, null deref, etc.).
>>
>> Personally I'd feel a lot better merging PRs I haven't thoroughly tested if
>> we had:
>>
>>   - Clang static analyzer during the build
>>   - A Git hook or manual use of clang-format or other formatter to prevent
>> the "extra white space" or "wrong comment style" type of issues that
>> commonly occur in PRs.
>>   - Ability for PRs to come with tests (requires testing support).
>>   - Linking in Clang's address & memory sanitizers while running all of the
>> tests.
>
> Geany is almost entirely an interactive application, so until
> interactive tests are possible I don't think technical tests like
> these will add a great deal to the committability of PRs.

If the tests just test functions, all it needs is to get Geany started 
up, then the tests can call the new/changed functions testing with 
different inputs and such. There are at least two PRs to do similar.

> Clangalizers and sanitizers and formatters won't tell you that the PR
> actually puts 'z' in the buffer instead of 'a'.
>

No, but they'll catch a number of runtime bugs that are often hard to 
identify upon basic code inspection or manual testing.

> Perhaps Columban knows more about using the accessibility framework
> for testing now Scintilla supports it?
>

There are several UI testing frameworks that work with GTK+, though I've 
not used any: autopilot, dogtail, and LDTP.

I don't think we really need fully automatic UI testing (seems like too 
much work), but we could get a long way just testing at the function 
level, ensuring functions uphold their contract and flexing them with 
unusual inputs. Making a testable function usually means writing it 
better too, avoiding global state and writing more "pure" functions, and 
making functions do one thing and not writing huge functions or many 
small functions.

Regards,
Matthew Brush