[Geany-Devel] [FT-plugins] Proposed "Features"

Jiří Techet techet at xxxxx
Wed Aug 31 15:15:49 UTC 2016

On Wed, Aug 31, 2016 at 1:22 PM, Thomas Martitz <kugel at rockbox.org> wrote:

> Am 31.08.2016 um 13:03 schrieb Jiří Techet:
>> On Tue, Aug 30, 2016 at 11:29 PM, Thomas Martitz <kugel at rockbox.org
>> <mailto:kugel at rockbox.org>> wrote:
>>     Am 30.08.2016 um 21:10 schrieb Jiří Techet:
>>                 Geany would then merge the tags, perhaps giving the plugin
>>                 ones more
>>                 weight, and store it in TM.
>>             I think you underestimate how many tags we're talking
>>         here. The
>>             example libclang ft-plugin would have to re-walk the
>>         entire AST
>>             (which is absolutely massive, particularly for C++),
>>         convert it to
>>             TM tag structures, and then Geany/TM would have to perform
>>         some
>>             merging logic, would would be more complicated than now if
>>         it was
>>             to support C++ properly, every single re-parse. My
>>         intuition tells
>>             me that just won't be fast enough, Clang already jumps through
>>             hoops and uses tricks to just build its own AST in-time.
>>         I think it would be a disaster performance-wise. The number of
>>         AST nodes can be easily 100x more than the amount of tags we
>>         have from ctags (we get a single tag for a function now and
>>         AST will contain complete tree for the function body) so just
>>         this might cost 100x more. In addition all the necessary
>>         copies to TM internal representation, having to maintain the
>>         tree structure (in TM we use GPtrArrays for everything which
>>         are very efficient and during tag merge we try to eliminate
>>         even pointer dereferences because those start getting
>>         expensive when managing many tags) etc.
>>     Let's not outright reject possible solutions based on performance
>>     assumptions and guestimations. Performance can be evaluated on an
>>     actual implementation. Until then it's simply an invalid argument
>>     for this discussion. But FWIW, I don't think performance is the
>>     driving aspect.
>> No, performance is a very valid point. Tag updates don't happen in a
>> background thread in Geany but rather on the main thread (and changing this
>> would require lots of modifications as neither ctags, nor TM nor Geany are
>> thread-safe) and all updates have to happen in a really short time period -
>> you cannot make the GUI freeze while the user is typing so you have 100ms
>> at most without any noticeable delay.
> I'm saying we can't evaluate performance at this time, because no
> implementation is available. So saying now "uhm I fear it might be too slow
> my guess is that the other one is 100x faster" is just a wild assumption
> that doesn't help except spraying FUD. And outright rejecting a proposal
> based on such assumptions is invalid.

100x more nodes of AST than the amount of tags we have isn't any wild
assumption - have a look at the AST picture here:


and the primitive code below to which it corresponds. The tree has 21
nodes. It's easy to imagine source files with 5x more code per function and
you are at the numbers I'm giving you - we generate 1 tag per function and
the corresponding AST you'll have to walk is 100x bigger.

> Please lets evaluate solutions, implement them, and then have an eye on
> performance.

I think I have evaluated this solution and it doesn't seem worth the
effort. Not only because the performance but also I don't know how to store
all the necessary information for any kind of language into TM so code
completion works for any language so that it's compatible with ctags and
any kind of AST. If you do, please tell us but not the way "One solution
can be to have tags hold sufficient information" because it doesn't tell

>>     What's needed from the AST is tags (as you and I mentioned
>>     elsewhere, AST is the complete code representation, so much more
>>     than what's required currently). Those can be extracted in one
>>     pass. I don't see that tags need to be converted back to the
>>     original AST.
>> As Matthew said, tags are insufficient for good autocompletion (e.g.
>> consider dynamic languages like Python where you have to infer variable
>> type from the right-hand side of an assignment and based on that generate
>> an autocompletion list).
> Tags with incomplete information are insufficient. One solution can be to
> have tags hold sufficient information.
> FWIW, unless you actually compile the stuff (for C++), any source code
> analysis will be insufficient for some cases. I was under the impression
> that we do *not* want mandatory compilation, which drags in build system
> implications, for stuff like auto completion.

libclang basically compiles it - it performs complete syntax analysis the
same way that would happen during compilation. It just skips code

> But if a smart ft-plugin does this I don't care. I only care about if/how
> Geany - and other non-ft-plugins - can use the data from such an analysis.
>>     For any successful plugin operation, the AST has to be generated
>>     (and probably re-generated regularly) and traversed at least once.
>>     Creating tags in a traversal that's happening done anyway probably
>>     isn't even going to add much overhead, if any.
>> But I assume AST is created in a background thread - as I said, all TM
>> code runs on the main thread and everything has to happen fast enough
>> between keystrokes so users don't experience any delays while typing.
> I don't care how the ft-plugin that provides tags to Geany generates it's
> internal AST. It can surely use a background thread. This is quite
> unrelated.

I was reacting to you where you said that AST has to be regenerated
regularly and that time spent on updating TM will be smaller. Yes, it will
be smaller, but it will have to be performed on the main thread instead of
in the background so the performance will be much more critical.

>>     As you say, we use a GPtrArray of tags because it's very efficient
>>     for sorting and merging. I think any ft-plugin will also have to
>>     sort (at least) for showing auto-completion and symbols tree - if
>>     it shows them itself. So it may even have to create such
>>     array/lists that TM uses anyway (you can't sort AST directly). So
>>     it might as well pass them to TM for sorting.
>> Whatever you do with tiny lists (say 1000 items) of tags which appear as
>> a result of autocompletion doesn't matter. I'm talking about workspace
>> array of tags which may contain all tags from a project with hundreds of
>> thousands or millions of tags and which have to be updated as the user
>> types and these updates have to be handled in a very efficient way.
> Wait, a AST comprising a million nodes doesn't result in a million
> autocompletion candidates. Anyway, how does the number relate to our
> discussion?

It appears to me you don't really read what I'm writing - I'm talking about
updating TM workspace tags which has to happen every time file is reparsed.
As I said, autocompletion list population performance doesn't matter much
because it will contain a lot less tags.

And by the way, projects with 1 000 000 tags will have more like 100 000
000 nodes in AST.

>>         And even if we did this, I don't know how we could handle ASTs
>>         of different languages in a generic way because these will
>>         differ significantly.
>>     One more time, seems I wasn't clear enough yet: I'm *not*
>>     suggesting that we create generic code inside Geany that handles
>>     any kind of AST. What I suggest is that plugins that use AST
>>     internally (or not) pass tag-like information to Geany, extracted
>>     and flattened from its internal AST (or whatever it uses). Then
>>     Geany and other plugins can use that information for their own
>>     purposes.
>> Again, for smart autocompletion you will need things from AST.
>> The biggest problem of current scope completion in Geany is the lack of
>> information about local variables which we might get with the new cxx ctags
>> parser and add them to TM. We'll get better results then but still
>> AST-based autocompletion will be able to do smarter things.
> Again, using AST is fine, within the ft-plugin. Use whatever you need from
> the AST for smart completion, and pass it in a generic format to Geany.
> This way Geany *will use* AST-based information just fine.

I'm ending the discussion here, don't expect more contributions on this
topic from me. You don't listen to what others are saying and keep
rejecting their arguments without giving any concrete proposals other than
"Use whatever you need" and "and pass it in a generic format to Geany"
which tell nothing. Continuing in this is a waste of time for me.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geany.org/pipermail/devel/attachments/20160831/79d5baf1/attachment-0001.html>

More information about the Devel mailing list