[Geany-Devel] Helping Geany move forward: testing

[Matthew Brush]

On 2017-04-28 06:35 PM, Lex Trotman wrote:
> ...
>>>
>>> Geany is almost entirely an interactive application, so until
>>> interactive tests are possible I don't think technical tests like
>>> these will add a great deal to the committability of PRs.
>>
>>
>> If the tests just test functions, all it needs is to get Geany started up,
>> then the tests can call the new/changed functions testing with different
>> inputs and such. There are at least two PRs to do similar.
>
> Sadly Geany isn't a pure functional program, most functions leave
> messy side-effects on global data, the Scintilla buffers :(
>
> So you need to be able to examine those.
>


You can, the tests are just regular extra functions called at runtime, 
you have access to all state that normal code does, it just makes it 
more trouble to setup/assert that state. When you have this in mind 
while writing a test for the new/changed function, you're more likely to 
make it more "pure" and single-task specific. The end result is better 
code and more testable code, which would gradually spread through parts 
of the codebase.

>>
>>> Clangalizers and sanitizers and formatters won't tell you that the PR
>>> actually puts 'z' in the buffer instead of 'a'.
>>>
>>
>> No, but they'll catch a number of runtime bugs that are often hard to
>> identify upon basic code inspection or manual testing.
>>
>>> Perhaps Columban knows more about using the accessibility framework
>>> for testing now Scintilla supports it?
>>>
>>
>> There are several UI testing frameworks that work with GTK+, though I've not
>> used any: autopilot, dogtail, and LDTP.
>>
>> I don't think we really need fully automatic UI testing (seems like too much
>> work), but we could get a long way just testing at the function level,
>> ensuring functions uphold their contract and flexing them with unusual
>> inputs. Making a testable function usually means writing it better too,
>> avoiding global state and writing more "pure" functions, and making
>> functions do one thing and not writing huge functions or many small
>> functions.
>
> We really NEED automatic UI testing and we NEED function unit testing,
> but realistically we are not going to get either.  If we don't have
> enough resources to just run and test PRs we don't have the resources
> to add these.
>

The contributors add the tests flexing the PR changes, giving the person 
merging the change more confidence and less reason to test every little 
corner case themselves, and are automated and repeatable to ensure the 
assumptions those tests make are not broken by other changes in the 
future. Instead of as the OP suggested, writing up a prose testing 
report by hand, they just write a test function that tests the 
assumptions they have checked, also showing missing assumptions.

Regards,
Matthew Brush