[Geany-Devel] [FT-plugins] Proposed "Features"

[Thomas Martitz]

Am 31.08.2016 um 13:03 schrieb Jiří Techet:
>
>
> On Tue, Aug 30, 2016 at 11:29 PM, Thomas Martitz <kugel at rockbox.org 
> <mailto:kugel at rockbox.org>> wrote:
>
>     Am 30.08.2016 um 21:10 schrieb Jiří Techet:
>
>                 Geany would then merge the tags, perhaps giving the plugin
>                 ones more
>                 weight, and store it in TM.
>
>
>             I think you underestimate how many tags we're talking
>         here. The
>             example libclang ft-plugin would have to re-walk the
>         entire AST
>             (which is absolutely massive, particularly for C++),
>         convert it to
>             TM tag structures, and then Geany/TM would have to perform
>         some
>             merging logic, would would be more complicated than now if
>         it was
>             to support C++ properly, every single re-parse. My
>         intuition tells
>             me that just won't be fast enough, Clang already jumps through
>             hoops and uses tricks to just build its own AST in-time.
>
>
>         I think it would be a disaster performance-wise. The number of
>         AST nodes can be easily 100x more than the amount of tags we
>         have from ctags (we get a single tag for a function now and
>         AST will contain complete tree for the function body) so just
>         this might cost 100x more. In addition all the necessary
>         copies to TM internal representation, having to maintain the
>         tree structure (in TM we use GPtrArrays for everything which
>         are very efficient and during tag merge we try to eliminate
>         even pointer dereferences because those start getting
>         expensive when managing many tags) etc.
>
>
>
>     Let's not outright reject possible solutions based on performance
>     assumptions and guestimations. Performance can be evaluated on an
>     actual implementation. Until then it's simply an invalid argument
>     for this discussion. But FWIW, I don't think performance is the
>     driving aspect.
>
>
> No, performance is a very valid point. Tag updates don't happen in a 
> background thread in Geany but rather on the main thread (and changing 
> this would require lots of modifications as neither ctags, nor TM nor 
> Geany are thread-safe) and all updates have to happen in a really 
> short time period - you cannot make the GUI freeze while the user is 
> typing so you have 100ms at most without any noticeable delay.

I'm saying we can't evaluate performance at this time, because no 
implementation is available. So saying now "uhm I fear it might be too 
slow my guess is that the other one is 100x faster" is just a wild 
assumption that doesn't help except spraying FUD. And outright rejecting 
a proposal based on such assumptions is invalid.

Please lets evaluate solutions, implement them, and then have an eye on 
performance.

>
>     What's needed from the AST is tags (as you and I mentioned
>     elsewhere, AST is the complete code representation, so much more
>     than what's required currently). Those can be extracted in one
>     pass. I don't see that tags need to be converted back to the
>     original AST.
>
>
> As Matthew said, tags are insufficient for good autocompletion (e.g. 
> consider dynamic languages like Python where you have to infer 
> variable type from the right-hand side of an assignment and based on 
> that generate an autocompletion list).

Tags with incomplete information are insufficient. One solution can be 
to have tags hold sufficient information.

FWIW, unless you actually compile the stuff (for C++), any source code 
analysis will be insufficient for some cases. I was under the impression 
that we do *not* want mandatory compilation, which drags in build system 
implications, for stuff like auto completion.

But if a smart ft-plugin does this I don't care. I only care about 
if/how Geany - and other non-ft-plugins - can use the data from such an 
analysis.

>
>     For any successful plugin operation, the AST has to be generated
>     (and probably re-generated regularly) and traversed at least once.
>     Creating tags in a traversal that's happening done anyway probably
>     isn't even going to add much overhead, if any.
>
>
> But I assume AST is created in a background thread - as I said, all TM 
> code runs on the main thread and everything has to happen fast enough 
> between keystrokes so users don't experience any delays while typing.


I don't care how the ft-plugin that provides tags to Geany generates 
it's internal AST. It can surely use a background thread. This is quite 
unrelated.


>
>     As you say, we use a GPtrArray of tags because it's very efficient
>     for sorting and merging. I think any ft-plugin will also have to
>     sort (at least) for showing auto-completion and symbols tree - if
>     it shows them itself. So it may even have to create such
>     array/lists that TM uses anyway (you can't sort AST directly). So
>     it might as well pass them to TM for sorting.
>
>
> Whatever you do with tiny lists (say 1000 items) of tags which appear 
> as a result of autocompletion doesn't matter. I'm talking about 
> workspace array of tags which may contain all tags from a project with 
> hundreds of thousands or millions of tags and which have to be updated 
> as the user types and these updates have to be handled in a very 
> efficient way.


Wait, a AST comprising a million nodes doesn't result in a million 
autocompletion candidates. Anyway, how does the number relate to our 
discussion?


>
>
>
>
>         And even if we did this, I don't know how we could handle ASTs
>         of different languages in a generic way because these will
>         differ significantly.
>
>
>     One more time, seems I wasn't clear enough yet: I'm *not*
>     suggesting that we create generic code inside Geany that handles
>     any kind of AST. What I suggest is that plugins that use AST
>     internally (or not) pass tag-like information to Geany, extracted
>     and flattened from its internal AST (or whatever it uses). Then
>     Geany and other plugins can use that information for their own
>     purposes.
>
>
> Again, for smart autocompletion you will need things from AST.
>
> The biggest problem of current scope completion in Geany is the lack 
> of information about local variables which we might get with the new 
> cxx ctags parser and add them to TM. We'll get better results then but 
> still AST-based autocompletion will be able to do smarter things.


Again, using AST is fine, within the ft-plugin. Use whatever you need 
from the AST for smart completion, and pass it in a generic format to 
Geany. This way Geany *will use* AST-based information just fine.

Best regards.