Hi All,
Related to my previous mail, and assuming it is somewhat reasonable, I would like to propose a list of initial features that will be useful to allow plugins to provide, in no particular order:
----
Syntax Highlighting -------------------
Most likely using an API based on/similar to Scintilla's "container lexers".
At the minimum, it could have a callback something like:
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*, guint start_pos, guint end_pos, gpointer user_data);
As with Scintilla's "container lexer", it would just tell the provider what and where to highlight. It might be pointless providing `end_pos` it could probably just highlight a whole line at time (maybe like Scintilla's 'style-needed' notification).
If the providers are to setup their own colour schemes and such, then it may be required to add a callback such as:
gboolean (*init_styles)(GeanyPlugin*, GeanyDocument*, gpointer user_data);
for the plugin to configure all the Scintilla styles as it wishes. This will probably require some kind of conflict avoidance scheme if it is to support multiple providers providing the same feature. Perhaps it could also pass the start style ID the provider can use, and it could tell Geany the maximum style that it ended up using.
Sidebar Symbol Tree -------------------
Could provide some API functions to populate the tree by plugins providing the needed information. The nesting could be handled by passing a scope path similar to GtkTreePath or TagManager's `::` delimited scope string, which would be parsed/expanded to apply to the right tree node.
The callback function for the provider might be like:
gboolean (*populate_symbols_tree)(GeanyPlugin*, GeanyDocument*, guint cursor_pos, gchar **current_node_path /* out */, gpointer user_data);
When the providers are called in to, they could use a function something like this:
void tagbar_add_node(const gchar *name, guint line, const gchar *signature, const gchar *scope_path, GeanySidebarIcon icon);
I haven't looked closely at the existing sidebar/symbols code yet, maybe it already provides such a function that could be used. It has also been mentioned that this could be done using the TagManager API. Plugins would walk their own internal ASTs or tag lists and build up the TM tag array, and Geany would use it for updating the symbols tree as it currently does (IIUC). I don't know much about TM so I can't really give an example callback for such an API.
Auto-Completion/Intellisense ----------------------------
This could be triggered at the exact same time in the same way it is now (and re-using the same preferences). It could call a callback function in the ft-plugins like:
gboolean (*complete_at)(GeanyPlugin *, GeanyDocument *, guint position, const gchar *partial_word, GPtrArray completion_list /* out */, gpointer user_data);
The `completion_list` would be filled in with whatever the provider thinks could be completed at `position`. It could be made more advanced in the future like allowing plugins to give icons, argument lists, documentation comment text, etc. For now it could just give strings to directly populate Scintilla's auto-complete listbox.
Calltips --------
To provide calltips, a provider could be called into like:
gboolean (*provide_calltips)(GeanyPlugin*, GeanyDocument*, guint position, const gchar *symbol, GPtrArray *overloads /* out */, gpointer user_data);
The `overloads` array would populated by the provider with the various overloads for the given `symbol` (function). As with auto-completion, at first this could just be a list of overloads represented as strings which directly populate Scintilla's calltip infos. This could be enhanced in the future to provide documentation comment text and placeholder insertion points into the editor in the future, but for now should be kept simple (see GtkSourceView, XCode, etc).
Go To Declaration/Definition ----------------------------
This could be either single callback passing a parameter to tell whether declaration or definition, or two separate callbacks. An example of the callbacks could be:
struct SourceLocation { gchar *filename; guint position; };
gboolean (*get_declaration_location)(GeanyPlugin*, GeanyDocument*, guint position, const gchar *symbol, GPtrArray *results, gpointer user_data);
and likewise for `get_definition_location`. The `results` array could be populated with `SourceLocation`s for each possible definition/declaration. For example there might be multiple declarations in different `#if` blocks you could jump to, for C-like languages. Geany already provides a UI popup list to pick where to jump, this should be re-used.
Build/Run Support -----------------
I haven't thought much on this yet, but in general there should be a way for ft-plugins to get called into when a the build/run features are activated.
Diagnostics -----------
I also haven't thought much on this yet. It needs to be possible for ft-plugins, after getting notified of build/run events, to be able to update diagnostics information.
Some of the diagnostics display might include:
- Scintilla marker margin icons, indicating the type of diagnostic info present for each line (could be more than one per line). - Scintilla indicators (little coloured squigglies) showing the specific range(s) of code where the diagnostics apply. - Scintilla annotations giving further details on request. - Tooltip messages when hovering over the marker margins or squigglied text giving further details. - Putting lines of diagnostics info in the msgwin compiler tab (already supported to some extent IIUC).
Refactoring -----------
This is just an idea that is probably long in the future, but ft-plugins could provide some common re-factorings (ex. rename, generate definition code, etc). I haven't thought on this much at all yet.
----
Almost surely I have left some out. What do you think?
Cheers, Matthew Brush
On Mon, Aug 29, 2016 at 5:14 AM, Matthew Brush mbrush@codebrainz.ca wrote:
Hi All,
Related to my previous mail, and assuming it is somewhat reasonable, I would like to propose a list of initial features that will be useful to allow plugins to provide, in no particular order:
Syntax Highlighting
Most likely using an API based on/similar to Scintilla's "container lexers".
At the minimum, it could have a callback something like:
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*, guint start_pos, guint end_pos, gpointer user_data);As with Scintilla's "container lexer", it would just tell the provider what and where to highlight. It might be pointless providing `end_pos` it could probably just highlight a whole line at time (maybe like Scintilla's 'style-needed' notification).
If the providers are to setup their own colour schemes and such, then it may be required to add a callback such as:
gboolean (*init_styles)(GeanyPlugin*, GeanyDocument*, gpointer user_data);for the plugin to configure all the Scintilla styles as it wishes. This will probably require some kind of conflict avoidance scheme if it is to support multiple providers providing the same feature. Perhaps it could also pass the start style ID the provider can use, and it could tell Geany the maximum style that it ended up using.
Sidebar Symbol Tree
Could provide some API functions to populate the tree by plugins providing the needed information. The nesting could be handled by passing a scope path similar to GtkTreePath or TagManager's `::` delimited scope string, which would be parsed/expanded to apply to the right tree node.
The callback function for the provider might be like:
gboolean (*populate_symbols_tree)(GeanyPlugin*, GeanyDocument*, guint cursor_pos, gchar **current_node_path /* out */, gpointer user_data);When the providers are called in to, they could use a function something like this:
void tagbar_add_node(const gchar *name, guint line, const gchar *signature, const gchar *scope_path, GeanySidebarIcon icon);I haven't looked closely at the existing sidebar/symbols code yet, maybe it already provides such a function that could be used. It has also been mentioned that this could be done using the TagManager API. Plugins would walk their own internal ASTs or tag lists and build up the TM tag array, and Geany would use it for updating the symbols tree as it currently does (IIUC). I don't know much about TM so I can't really give an example callback for such an API.
Auto-Completion/Intellisense
This could be triggered at the exact same time in the same way it is now (and re-using the same preferences). It could call a callback function in the ft-plugins like:
gboolean (*complete_at)(GeanyPlugin *, GeanyDocument *, guint position, const gchar *partial_word, GPtrArray completion_list /* out */, gpointer user_data);The `completion_list` would be filled in with whatever the provider thinks could be completed at `position`. It could be made more advanced in the future like allowing plugins to give icons, argument lists, documentation comment text, etc. For now it could just give strings to directly populate Scintilla's auto-complete listbox.
Calltips
To provide calltips, a provider could be called into like:
gboolean (*provide_calltips)(GeanyPlugin*, GeanyDocument*, guint position, const gchar *symbol, GPtrArray *overloads /* out */, gpointer user_data);The `overloads` array would populated by the provider with the various overloads for the given `symbol` (function). As with auto-completion, at first this could just be a list of overloads represented as strings which directly populate Scintilla's calltip infos. This could be enhanced in the future to provide documentation comment text and placeholder insertion points into the editor in the future, but for now should be kept simple (see GtkSourceView, XCode, etc).
Go To Declaration/Definition
This could be either single callback passing a parameter to tell whether declaration or definition, or two separate callbacks. An example of the callbacks could be:
struct SourceLocation { gchar *filename; guint position; }; gboolean (*get_declaration_location)(GeanyPlugin*, GeanyDocument*, guint position, const gchar *symbol, GPtrArray *results, gpointer user_data);and likewise for `get_definition_location`. The `results` array could be populated with `SourceLocation`s for each possible definition/declaration. For example there might be multiple declarations in different `#if` blocks you could jump to, for C-like languages. Geany already provides a UI popup list to pick where to jump, this should be re-used.
Build/Run Support
I haven't thought much on this yet, but in general there should be a way for ft-plugins to get called into when a the build/run features are activated.
Diagnostics
I also haven't thought much on this yet. It needs to be possible for ft-plugins, after getting notified of build/run events, to be able to update diagnostics information.
Some of the diagnostics display might include:
- Scintilla marker margin icons, indicating the type of diagnostic info
present for each line (could be more than one per line).
- Scintilla indicators (little coloured squigglies) showing the specific
range(s) of code where the diagnostics apply.
- Scintilla annotations giving further details on request.
- Tooltip messages when hovering over the marker margins or squigglied
text giving further details.
- Putting lines of diagnostics info in the msgwin compiler tab (already
supported to some extent IIUC).
Refactoring
This is just an idea that is probably long in the future, but ft-plugins could provide some common re-factorings (ex. rename, generate definition code, etc). I haven't thought on this much at all yet.
Almost surely I have left some out. What do you think?
Cheers, Matthew Brush _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Hi Matthew,
sounds good. This is much more lightweight than how #1195 and various other discussions sounded, I'm happy :-).
All the described functions look good to me in principle except the sidebar symbol tree one. The current code is quite complex (I was fixing some minor issues there and spent a lot of time thinking if it could be simplified but I don't think so). The problem is this: if a file is reparsed and you get new symbols, if you just cleared the tree and repopulated it with the new tags, the tree would jump to the beginning no matter where the scrollbar was before. So what the code does now is it compares the old and new tags, removes the tags that aren't present in the new tags array and inserts new tags which weren't present in the old tag array to the right position. If you have a look at the code, this isn't probably something you want to write in a plugin :-). So a plugin should provide the full list of new tags so the tree can do all this diffing and updating.
Which brings me to a question - do you plan to generate TMTag(s) and feed them to the tag manager instead of the ctags ones? It shouldn't be that hard and if you do this, you could have the sidebar symbols updated for free.
These fields in TMTag are unused, don't waste time filling them:
gboolean local; /**< Is the tag of local scope */ guint pointerOrder; char *inheritance; /**< Parent classes */ char access; /**< Access type (public/protected/private/etc.) */ char impl; /**< Implementation (e.g. virtual) */
Cheers,
Jiri
Le 29/08/2016 à 10:04, Jiří Techet a écrit :
[…]
sounds good. This is much more lightweight than how #1195 and various other discussions sounded, I'm happy :-).
Agreed :) I was a little afraid of seeing a proposal introducing a gazillion GObject interfaces and GIO extension points ^^
All the described functions look good to me in principle except the sidebar symbol tree one. The current code is quite complex (I was fixing some minor issues there and spent a lot of time thinking if it could be simplified but I don't think so). The problem is this: if a file is reparsed and you get new symbols, if you just cleared the tree and repopulated it with the new tags, the tree would jump to the beginning no matter where the scrollbar was before. So what the code does now is it compares the old and new tags, removes the tags that aren't present in the new tags array and inserts new tags which weren't present in the old tag array to the right position. If you have a look at the code, this isn't probably something you want to write in a plugin :-). So a plugin should provide the full list of new tags so the tree can do all this diffing and updating.
Agreed. And the tree updating avoids scroll movements, flickering and also proven significantly faster than re-building the whole tree (well, than the previous code at least).
Which brings me to a question - do you plan to generate TMTag(s) and feed them to the tag manager instead of the ctags ones? It shouldn't be that hard and if you do this, you could have the sidebar symbols updated for free.
I don't know if plugins should fill the TagManager with extra tags, but I agree that plugins should probably use the TMTag structure to pass Tag-like data to Geany. For example, for symbols tree, autocompletion suggestions and calltips: instead of providing a list of strings, provide a list of TMTags. As I see it, a TMTag structure contains everything useful for the current feature set (and could be extended), and are a fairly canonical representation of it. A plugin could create temporary TMTags just to give to Geany, or maintain its own list (or feed TagManager, if we wanted), but anyway just pass the ones it want to the various APIs.
TMTags contain name, scope (useful for symbol tree and currently calltips), type (useful for icons at least), signature, etc. All we need in various places. And a plugin could leave many fields empty if it doesn't care about them, not many fields are actually required.
Regards, Colomban
Am 30.08.2016 um 15:38 schrieb Colomban Wendling:
Which brings me to a question - do you plan to generate TMTag(s) and feed them to the tag manager instead of the ctags ones? It shouldn't be that hard and if you do this, you could have the sidebar symbols updated for free.
I don't know if plugins should fill the TagManager with extra tags, but I agree that plugins should probably use the TMTag structure to pass Tag-like data to Geany. For example, for symbols tree, autocompletion suggestions and calltips: instead of providing a list of strings, provide a list of TMTags. As I see it, a TMTag structure contains everything useful for the current feature set (and could be extended), and are a fairly canonical representation of it. A plugin could create temporary TMTags just to give to Geany, or maintain its own list (or feed TagManager, if we wanted), but anyway just pass the ones it want to the various APIs.
TMTags contain name, scope (useful for symbol tree and currently calltips), type (useful for icons at least), signature, etc. All we need in various places. And a plugin could leave many fields empty if it doesn't care about them, not many fields are actually required.
I think we're on the same page here. I'm also suggesting that we build a framework which passes tags (or tag-like as you name it) from plugins to Geany, and Geany uses that to implement features X and Y. If the TMTag structure is currently insufficient then, well, just extend it as needed.
This is flexible and elegant, and allows to develop new feature (inside Geany or non-ft-plugins) without changing all the ft-plugins as well.
Best regards.
On 2016-08-30 06:38 AM, Colomban Wendling wrote:
Le 29/08/2016 à 10:04, Jiří Techet a écrit :
[…]
Which brings me to a question - do you plan to generate TMTag(s) and feed them to the tag manager instead of the ctags ones? It shouldn't be that hard and if you do this, you could have the sidebar symbols updated for free.
I don't know if plugins should fill the TagManager with extra tags, but I agree that plugins should probably use the TMTag structure to pass Tag-like data to Geany. For example, for symbols tree, autocompletion suggestions and calltips: instead of providing a list of strings, provide a list of TMTags. As I see it, a TMTag structure contains everything useful for the current feature set (and could be extended), and are a fairly canonical representation of it. A plugin could create temporary TMTags just to give to Geany, or maintain its own list (or feed TagManager, if we wanted), but anyway just pass the ones it want to the various APIs.
TMTags contain name, scope (useful for symbol tree and currently calltips), type (useful for icons at least), signature, etc. All we need in various places. And a plugin could leave many fields empty if it doesn't care about them, not many fields are actually required.
I feel like it would be easier to provide a small API to fill the tree (and even provide this - surprisingly to me - complicated code to make the treeview do what it's main purpose is), and that filling in TM tags might be tedious API-wise, but I admittedly don't know enough about TM and the sidebar code to make this judgment with full confidence.
It seems like you, Jiří and Thomas all agree (to varying degrees) that it makes sense to re-use TM for some stuff like this, so it's fine with me. When I update the Github tracker issue with the results of these discussions, I will include this usage of TM tags (at least).
Cheers, Matthew Brush
Am 29.08.2016 um 05:14 schrieb Matthew Brush:
Hi All,
Related to my previous mail, and assuming it is somewhat reasonable, I would like to propose a list of initial features that will be useful to allow plugins to provide, in no particular order:
I disagree with this approach. This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases.
I thought we agreed that plugin should simply provide tags to Geany/TM and then Geany uses them for various purposes as it sees fit (possibly depending on user preferences), even if it means to extend TM in a few places.
Sorry, I will not support the approach you are suggesting, until it's clear to me that it's not doable otherwise. Instead, we should inspect each use case and locate exact requirements for "tag passing".
PS: highlighting and tags are generally separated topics. With custom filetypes we can have highlighting even if we don't have a parser. I'd rather not mix highlighting in this TM-related topics. Though I could imagine we could make it work with some magic tags that provide keywords to be highlighted, or just simply make use of dynamic lexers.
Best regards.
This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases.
The point of this proposal is to change and add use-cases that are not currently possible with the current plugin API. But instead of each use-case generating its own piece of API and its own infrastructure, the point of the FT-plugins proposal is to provide a common infrastructure and approach for all filetype specific use-cases, those needed for currently suggested uses, indentation, clang based styling and symbols, and as framework for future use-cases we either havn't thought of, or havn't a concrete intention to add immediately.
I thought we agreed that plugin should simply provide tags to Geany/TM
This proposal is about many types of filetype specific functionality, not just tags. Tagmanager will not help in any way with indenting Haskell, or even C++.
I'd rather not mix highlighting in this TM-related topics.
It appears you have misunderstood the purpose of the proposal, or I have.
It is my understanding that, as noted above, the proposal is to allow plugins to provide filetype specific capabilities for many areas, "features" in Matthews parlance. Its not intending to change the existing plugin API but to extend it to support functionality that cannot be performed at the moment.
An example is Matthews attempt at using libclang in a plugin, which provides, highlighting, symbol lookups, formatting etc with much greater accuracy than the existing facilities in Geany can do. But at the moment there is no facility in Geany for any of that that information to be injected for any filetype, or without forcing it to be limited by Geany built-in capabilities.
Jiri has for example noted he doesn't want to be built into Geany the type of project management capability needed to allow accurate enough includes to be determined to allow accurate symbols. Thats fine, but at the moment its also not possible for anyone to add it in a plugin either. If Geany is to remain closed to such plugins it will remain "all languages are poorly handled C" and slip further behind other tools for newer languages that become more and more "not C". And I for one would be sad to see that happen simply because of a refusal to make Geany open enough.
Cheers Lex
Am 29.08.2016 um 14:23 schrieb Lex Trotman:
This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases. The point of this proposal is to change and add use-cases that are not currently possible with the current plugin API. But instead of each use-case generating its own piece of API and its own infrastructure, the point of the FT-plugins proposal is to provide a common infrastructure and approach for all filetype specific use-cases, those needed for currently suggested uses, indentation, clang based styling and symbols, and as framework for future use-cases we either havn't thought of, or havn't a concrete intention to add immediately.
I thought we agreed that plugin should simply provide tags to Geany/TM
This proposal is about many types of filetype specific functionality, not just tags. Tagmanager will not help in any way with indenting Haskell, or even C++.
4 of 5 of the proposed features are strictly tag-related. And Geany can do all of them already, it's just that the current implementation leaves things to be desired so there is the idea to let plugins improve upon them.
I disagree with the proposed solution for those 4, because they are offloading logic on a per feature basis to plugins, only because Geany isn't capable at the moment. If Geany was capable, then there could be 1 solution for the 4 features and less complexity in each plugin (and we know the quality of plugins varies a lot so they should have little complexity as possible).
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.
Best regards.
On 2016-08-29 05:38 AM, Thomas Martitz wrote:
Am 29.08.2016 um 14:23 schrieb Lex Trotman:
This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases. The point of this proposal is to change and add use-cases that are not currently possible with the current plugin API. But instead of each use-case generating its own piece of API and its own infrastructure, the point of the FT-plugins proposal is to provide a common infrastructure and approach for all filetype specific use-cases, those needed for currently suggested uses, indentation, clang based styling and symbols, and as framework for future use-cases we either havn't thought of, or havn't a concrete intention to add immediately.
I thought we agreed that plugin should simply provide tags to Geany/TM
This proposal is about many types of filetype specific functionality, not just tags. Tagmanager will not help in any way with indenting Haskell, or even C++.
4 of 5 of the proposed features are strictly tag-related. And Geany can do all of them already, it's just that the current implementation leaves things to be desired so there is the idea to let plugins improve upon them.
I disagree with the proposed solution for those 4, because they are offloading logic on a per feature basis to plugins, only because Geany isn't capable at the moment. If Geany was capable, then there could be 1 solution for the 4 features and less complexity in each plugin (and we know the quality of plugins varies a lot so they should have little complexity as possible).
Using TagManager would likely add more complexity to the plugins. Using the libclang example, it already has its own internal representation of "tags" (ie. the AST). Making plugins munge their internal representation to convert/conform to TagManager's would add extra burden.
For example "go to definition/declaration" is a simple task with libclang, you would just get the source location and pass back to Geany the filename and position and it could go there easily, without any need for TagManager as a middle-man. Likewise, libclang has a function to return valid completions for a given position in the source code, passing a list of strings for Geany to show in the Scintilla auto-complete list would be a simple matter of building a string array. Using tag manager would require building a tags array that probably wouldn't/shouldn't become part of any larger tags array (workspace/global/whatever) and then getting Geany to show that tags array without messing with it (ordering, etc).
Another issue that might arise is that TagManager might very well choke if it had to deal with every single tag/symbol in the entire source (make a "hello world" in C++ using iostream and look at the preprocessed source, for example). The number of tags it would need to store/track would way more than double when you consider local variables, parameters, inner-classes, etc. I have no clue if TM is actually able to handle this at all, efficiently enough. Ontop of that, all this information is stored twice in memory, once in the libclang AST and once in TagManager tag arrays. This could result in quite some additional memory overhead.
Jiří gives an example (the symbols sidebar) where this may be warranted and ultimately cause less work, since the sidebar updating code is apparently quite complicated. I still need to study this code to understand why so much effort is needed to keep the symbol tree at the same relative scroll location when updated.
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.
I'm not strictly opposed to wrangling stuff through TagManager. Could you provide example hooks showing the kind of interface you would thinking about for the mentioned features? Also in which cases do you expect TagManager is not up to the task and would need to be modified?
Cheers, Matthew Brush
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.
I'm personally very happy that Matthew decided to go the way he proposed instead of doing what you propose. As I said in one of the posts here
https://github.com/geany/geany/pull/1187
it's not a matter of how "advanced" the tags are but that the tags are represented in a completely different way - tree vs list. There's info in the tree that cannot be stored in the list and similarly ctags list cannot be converted to AST. You can start doing some crazy things like serializing AST to a list and then deserializing it to a tree or having parallel AST-TM and list-TM but things are complicated enough in TM already and merging ASTs during source file updates would be really complicated. Also performance would suffer because operations on GPtrArrays are really fast as they don't require pointer dereferencing. In addition I think each library will have its own partially incompatible AST representation and different info will be needed e.g. for autocompletion and it will be really hard to find something generic enough.
So big +1 for Matthew's approach from me. What might be good though is filling the "simple" TMTags with "advanced" libclang or any other tags into the existing TM. Some info will be lost on the way but this info will be sufficient for some things like the symbol tree or your plugin.
Jiri
Am 29.08.2016 um 17:05 schrieb Jiří Techet:
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.I'm personally very happy that Matthew decided to go the way he proposed instead of doing what you propose. As I said in one of the posts here
https://github.com/geany/geany/pull/1187
it's not a matter of how "advanced" the tags are but that the tags are represented in a completely different way - tree vs list. There's info in the tree that cannot be stored in the list and similarly ctags list cannot be converted to AST. You can start doing some crazy things like serializing AST to a list and then deserializing it to a tree or having parallel AST-TM and list-TM but things are complicated enough in TM already and merging ASTs during source file updates would be really complicated. Also performance would suffer because operations on GPtrArrays are really fast as they don't require pointer dereferencing. In addition I think each library will have its own partially incompatible AST representation and different info will be needed e.g. for autocompletion and it will be really hard to find something generic enough.
1) trees can be stored in flat list simply by storing the tree path in each element (like GtkTreeModel). This is not crazy or complex. It's even readily available in the scope string. 2) I don't see why the AST needs to be recoverable from a TM tag list. You only need to map from X to TM, so losing information is OK as long as it's sufficient for the function of current features/use cases (for future features TM tag list may or may not needed to be enhanced).
Please show me how the proposed tag-related features require advanced tag management that isn't possible with TM if tags would be supplemented or completely provided by a plugin.
What might be good though is filling the "simple" TMTags with "advanced" libclang or any other tags into the existing TM. Some info will be lost on the way but this info will be sufficient for some things like the symbol tree or your plugin.
Right, this is basically what I'm looking at. I can't see how we need anything more at this point. And yes, I do care that Geany itself and plugins (including mine) have a unified interface to access all tags, regardless of who provides them. Matthews proposal prohibits this.
There is also another aspect about the proposal that worries me: a plugin shall provide N features for M languages. And X plugins might be compete (not even considering the desire that plugins can build upon each other). This means (slightly exaggerated) that N*M*X possibilities have to be managed by Geany's conflict resolution. I don't want to implement that. It seems much simpler to me to collect tags from plugins, merge them (maybe throw out duplicates) and pass them to the actual feature code all within Geany.
What worries me is that we jumped from mere brainstorming to a relatively concrete proposal, without evaluating requirements or any other research. Or was this evaluation just invisible to me?
Best regards.
On 2016-08-29 03:17 PM, Thomas Martitz wrote:
Am 29.08.2016 um 17:05 schrieb Jiří Techet:
[...]
There is also another aspect about the proposal that worries me: a plugin shall provide N features for M languages. And X plugins might be compete (not even considering the desire that plugins can build upon each other). This means (slightly exaggerated) that N*M*X possibilities have to be managed by Geany's conflict resolution. I don't want to implement that. It seems much simpler to me to collect tags from plugins, merge them (maybe throw out duplicates) and pass them to the actual feature code all within Geany.
In principle it's not that hard to manage, as mentioned in my "Proposed Design" message, Geany just needs to keep the providers in a list and the callbacks work like GTK/GDK callbacks (and some in Geany) where the callback's return value determines whether the next provider is called or not. In that message I attached a mockup of a kind of UI that could be used to allow users absolute control, and for Geany it's just a (set of) ordered lists.
What worries me is that we jumped from mere brainstorming to a relatively concrete proposal, without evaluating requirements or any other research. Or was this evaluation just invisible to me?
I evaluated and experimented with several different approaches and discussed various details with some of the main developers (including you) on IRC. Based on the way that in my opinion, as someone who has tried and failed to implement the needed features in the past, and as a Geany developer, I recommended a proposed design for further input. And here we are :)
As I asked in an earlier message, I'd be interested if you could provide some more concrete examples of what you were thinking with using TM, which would accomplish the goals mentioned in the Github Issue and fleshed-out more in the top of this thread.
Cheers, Matthew Brush
Am 30.08.2016 um 01:53 schrieb Matthew Brush:
On 2016-08-29 03:17 PM, Thomas Martitz wrote:
Am 29.08.2016 um 17:05 schrieb Jiří Techet:
[...]
There is also another aspect about the proposal that worries me: a plugin shall provide N features for M languages. And X plugins might be compete (not even considering the desire that plugins can build upon each other). This means (slightly exaggerated) that N*M*X possibilities have to be managed by Geany's conflict resolution. I don't want to implement that. It seems much simpler to me to collect tags from plugins, merge them (maybe throw out duplicates) and pass them to the actual feature code all within Geany.
In principle it's not that hard to manage, as mentioned in my "Proposed Design" message, Geany just needs to keep the providers in a list and the callbacks work like GTK/GDK callbacks (and some in Geany) where the callback's return value determines whether the next provider is called or not. In that message I attached a mockup of a kind of UI that could be used to allow users absolute control, and for Geany it's just a (set of) ordered lists.
You say this should be easy, I say I expect it to be complicated. This is quite the same (just the other way around) with my suggestion to just pass tags to Geany. There you keep claiming that it'll be a massive change why I expect it to be relatively easy to do. At least not harder than changing 6+ core parts in Geany to execute plugin callbacks and make something useful from the results.
What worries me is that we jumped from mere brainstorming to a relatively concrete proposal, without evaluating requirements or any other research. Or was this evaluation just invisible to me?
I evaluated and experimented with several different approaches and discussed various details with some of the main developers (including you) on IRC. Based on the way that in my opinion, as someone who has tried and failed to implement the needed features in the past, and as a Geany developer, I recommended a proposed design for further input. And here we are :)
Please show me the point in the IRC logs where I agreed with your approach. In fact, I can't remember discussing ft-plugins with you on IRC at all (I just asked at one point how libclang works in more detail).
Your evaluation is still invisible to me.
As I asked in an earlier message, I'd be interested if you could provide some more concrete examples of what you were thinking with using TM, which would accomplish the goals mentioned in the Github Issue and fleshed-out more in the top of this thread.
Essentially I'd propose a signal that's emitted when Geany begins (re-)parsing a file (I believe it does this after each keystroke, after a debouncing timeout). Plugins could connect to that signal, perhaps parse the file again, using their special, language specific library code, and pass the tags they've got to Geany using a to-be-designed API function (probably involving TMTag and TMSourceFile). Alternatively, the plugin signal handlers could run before Geany's own one, potentially blocking Geany from parsing the file at all. Or both approaches.
Geany would then merge the tags, perhaps giving the plugin ones more weight, and store it in TM.
Then the symbol tree, completion, other plugins can use the tags and make their stuff.
For highlighting, I would really try to use Scintilla dynamic lexers. Otherwise we're going to have trouble with inconsistent editor widgets and non-working color schemes.
Best regards
On 2016-08-30 08:51 AM, Thomas Martitz wrote:
Am 30.08.2016 um 01:53 schrieb Matthew Brush:
On 2016-08-29 03:17 PM, Thomas Martitz wrote:
Am 29.08.2016 um 17:05 schrieb Jiří Techet:
[...]
There is also another aspect about the proposal that worries me: a plugin shall provide N features for M languages. And X plugins might be compete (not even considering the desire that plugins can build upon each other). This means (slightly exaggerated) that N*M*X possibilities have to be managed by Geany's conflict resolution. I don't want to implement that. It seems much simpler to me to collect tags from plugins, merge them (maybe throw out duplicates) and pass them to the actual feature code all within Geany.
In principle it's not that hard to manage, as mentioned in my "Proposed Design" message, Geany just needs to keep the providers in a list and the callbacks work like GTK/GDK callbacks (and some in Geany) where the callback's return value determines whether the next provider is called or not. In that message I attached a mockup of a kind of UI that could be used to allow users absolute control, and for Geany it's just a (set of) ordered lists.
You say this should be easy, I say I expect it to be complicated. This is quite the same (just the other way around) with my suggestion to just pass tags to Geany. There you keep claiming that it'll be a massive change why I expect it to be relatively easy to do. At least not harder than changing 6+ core parts in Geany to execute plugin callbacks and make something useful from the results.
I've tinkered with implementations, it's generally as easy as walking a list in a loop, calling a function and breaking early if the function returns `FALSE`.
I don't think I claimed it will be a massive change, I think I just said it will add more complexity to ft-plugins, as opposed to less, as you claimed.
Since the very first response I made to you about TM, I've been open to the idea. I raised a few concerns which you never responded to, namely that it may not be possible for TM to handle all the tags in Clang's AST (it has a very rich and complex AST, heavily optimized by some of the smartest people in the C++ world). Also the AST is HUGE and representing it twice in memory would at least (if not more than) double the memory footprint.
What worries me is that we jumped from mere brainstorming to a relatively concrete proposal, without evaluating requirements or any other research. Or was this evaluation just invisible to me?
I evaluated and experimented with several different approaches and discussed various details with some of the main developers (including you) on IRC. Based on the way that in my opinion, as someone who has tried and failed to implement the needed features in the past, and as a Geany developer, I recommended a proposed design for further input. And here we are :)
Please show me the point in the IRC logs where I agreed with your approach. In fact, I can't remember discussing ft-plugins with you on IRC at all (I just asked at one point how libclang works in more detail).
I never said I proposed a design on IRC, that's what the "Proposed Design" email was about. I discussed approaches and details with some of the developers. You and I talked about whether TM might be up to the job of representing scope, for example (and you pointed out that it can, crudely, by using strings rather than parent/child relationships).
Your evaluation is still invisible to me.
I evaluated the various approaches myself and by discussing specific topics with some devs lately and previously when we talked about ft-plugins so that I would be able to propose a design on the mailing list, and here we are discussing that. I don't see the problem.
As I asked in an earlier message, I'd be interested if you could provide some more concrete examples of what you were thinking with using TM, which would accomplish the goals mentioned in the Github Issue and fleshed-out more in the top of this thread.
Essentially I'd propose a signal that's emitted when Geany begins (re-)parsing a file (I believe it does this after each keystroke, after a debouncing timeout). Plugins could connect to that signal, perhaps parse the file again, using their special, language specific library code, and pass the tags they've got to Geany using a to-be-designed API function (probably involving TMTag and TMSourceFile). Alternatively, the plugin signal handlers could run before Geany's own one, potentially blocking Geany from parsing the file at all. Or both approaches.
Unrelated to whether to use TM or not, I think you're right that a hook to tell ft-plugins to re-parse would be useful, otherwise plugins will all have to implement that logic/signal connections themselves.
Geany would then merge the tags, perhaps giving the plugin ones more weight, and store it in TM.
I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time.
Then the symbol tree, completion, other plugins can use the tags and make their stuff.
For highlighting, I would really try to use Scintilla dynamic lexers. Otherwise we're going to have trouble with inconsistent editor widgets and non-working color schemes.
I have tried that, I can try and find the code if I still have it. The problem I have with it is that it adds a whole other framework (ILexer) and language (C++) that has to be used, and it's a lot more involved just to perform basically the same task. In both cases ft-plugins could either provide their own states or re-use those from an existing lexer built-into Geany, so I don't think container vs dynamic lexer will have any bearing on the inconsistency of colour schemes.
Cheers, Matthew Brush
On Tue, Aug 30, 2016 at 6:41 PM, Matthew Brush mbrush@codebrainz.ca wrote:
On 2016-08-30 08:51 AM, Thomas Martitz wrote:
Am 30.08.2016 um 01:53 schrieb Matthew Brush:
On 2016-08-29 03:17 PM, Thomas Martitz wrote:
Am 29.08.2016 um 17:05 schrieb Jiří Techet:
[...]
There is also another aspect about the proposal that worries me: a plugin shall provide N features for M languages. And X plugins might be compete (not even considering the desire that plugins can build upon each other). This means (slightly exaggerated) that N*M*X possibilities have to be managed by Geany's conflict resolution. I don't want to implement that. It seems much simpler to me to collect tags from plugins, merge them (maybe throw out duplicates) and pass them to the actual feature code all within Geany.
In principle it's not that hard to manage, as mentioned in my "Proposed Design" message, Geany just needs to keep the providers in a list and the callbacks work like GTK/GDK callbacks (and some in Geany) where the callback's return value determines whether the next provider is called or not. In that message I attached a mockup of a kind of UI that could be used to allow users absolute control, and for Geany it's just a (set of) ordered lists.
You say this should be easy, I say I expect it to be complicated. This is quite the same (just the other way around) with my suggestion to just pass tags to Geany. There you keep claiming that it'll be a massive change why I expect it to be relatively easy to do. At least not harder than changing 6+ core parts in Geany to execute plugin callbacks and make something useful from the results.
I've tinkered with implementations, it's generally as easy as walking a list in a loop, calling a function and breaking early if the function returns `FALSE`.
I don't think I claimed it will be a massive change, I think I just said it will add more complexity to ft-plugins, as opposed to less, as you claimed.
Since the very first response I made to you about TM, I've been open to the idea. I raised a few concerns which you never responded to, namely that it may not be possible for TM to handle all the tags in Clang's AST (it has a very rich and complex AST, heavily optimized by some of the smartest people in the C++ world). Also the AST is HUGE and representing it twice in memory would at least (if not more than) double the memory footprint.
What worries me is that we jumped from mere brainstorming to a
relatively concrete proposal, without evaluating requirements or any other research. Or was this evaluation just invisible to me?
I evaluated and experimented with several different approaches and discussed various details with some of the main developers (including you) on IRC. Based on the way that in my opinion, as someone who has tried and failed to implement the needed features in the past, and as a Geany developer, I recommended a proposed design for further input. And here we are :)
Please show me the point in the IRC logs where I agreed with your approach. In fact, I can't remember discussing ft-plugins with you on IRC at all (I just asked at one point how libclang works in more detail).
I never said I proposed a design on IRC, that's what the "Proposed Design" email was about. I discussed approaches and details with some of the developers. You and I talked about whether TM might be up to the job of representing scope, for example (and you pointed out that it can, crudely, by using strings rather than parent/child relationships).
Your evaluation is still invisible to me.
I evaluated the various approaches myself and by discussing specific topics with some devs lately and previously when we talked about ft-plugins so that I would be able to propose a design on the mailing list, and here we are discussing that. I don't see the problem.
As I asked in an earlier message, I'd be interested if you could provide some more concrete examples of what you were thinking with using TM, which would accomplish the goals mentioned in the Github Issue and fleshed-out more in the top of this thread.
Essentially I'd propose a signal that's emitted when Geany begins (re-)parsing a file (I believe it does this after each keystroke, after a debouncing timeout). Plugins could connect to that signal, perhaps parse the file again, using their special, language specific library code, and pass the tags they've got to Geany using a to-be-designed API function (probably involving TMTag and TMSourceFile). Alternatively, the plugin signal handlers could run before Geany's own one, potentially blocking Geany from parsing the file at all. Or both approaches.
Unrelated to whether to use TM or not, I think you're right that a hook to tell ft-plugins to re-parse would be useful, otherwise plugins will all have to implement that logic/signal connections themselves.
Geany would then merge the tags, perhaps giving the plugin ones more
weight, and store it in TM.
I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time.
I think it would be a disaster performance-wise. The number of AST nodes can be easily 100x more than the amount of tags we have from ctags (we get a single tag for a function now and AST will contain complete tree for the function body) so just this might cost 100x more. In addition all the necessary copies to TM internal representation, having to maintain the tree structure (in TM we use GPtrArrays for everything which are very efficient and during tag merge we try to eliminate even pointer dereferences because those start getting expensive when managing many tags) etc.
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly.
Anyway, if needed we can always add more elements to the TMTag structure so plugins can add some more information.
Cheers,
Jiri
Am 30.08.2016 um 21:10 schrieb Jiří Techet:
Geany would then merge the tags, perhaps giving the plugin ones more weight, and store it in TM. I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time.I think it would be a disaster performance-wise. The number of AST nodes can be easily 100x more than the amount of tags we have from ctags (we get a single tag for a function now and AST will contain complete tree for the function body) so just this might cost 100x more. In addition all the necessary copies to TM internal representation, having to maintain the tree structure (in TM we use GPtrArrays for everything which are very efficient and during tag merge we try to eliminate even pointer dereferences because those start getting expensive when managing many tags) etc.
Let's not outright reject possible solutions based on performance assumptions and guestimations. Performance can be evaluated on an actual implementation. Until then it's simply an invalid argument for this discussion. But FWIW, I don't think performance is the driving aspect.
What's needed from the AST is tags (as you and I mentioned elsewhere, AST is the complete code representation, so much more than what's required currently). Those can be extracted in one pass. I don't see that tags need to be converted back to the original AST.
For any successful plugin operation, the AST has to be generated (and probably re-generated regularly) and traversed at least once. Creating tags in a traversal that's happening done anyway probably isn't even going to add much overhead, if any.
As you say, we use a GPtrArray of tags because it's very efficient for sorting and merging. I think any ft-plugin will also have to sort (at least) for showing auto-completion and symbols tree - if it shows them itself. So it may even have to create such array/lists that TM uses anyway (you can't sort AST directly). So it might as well pass them to TM for sorting.
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly.
One more time, seems I wasn't clear enough yet: I'm *not* suggesting that we create generic code inside Geany that handles any kind of AST. What I suggest is that plugins that use AST internally (or not) pass tag-like information to Geany, extracted and flattened from its internal AST (or whatever it uses). Then Geany and other plugins can use that information for their own purposes.
Anyway, if needed we can always add more elements to the TMTag structure so plugins can add some more information.
Yes. I do want to be able to implement new tag-related features in Geany or other plugins. So if an ft-plugin improves tags over Geany's builtin abilities, then these should become generally available.
Otherwise we get into a situation where core features of Geany become exclusive to individual plugins and cannot be improved upon again (get stuck with a unmaintained plugin that isn't updated to newer Geany features, anyone?).
Considering the poor quality and orphaned status of some G-P plugins, I wouldn't want to give individual plugins exclusive access to otherwise appreciated features which are mostly working in Geany core as of today. I'd rather not use ft-plugins at all if they provoke such a situation.
Best regards.
On 2016-08-30 02:29 PM, Thomas Martitz wrote:
Am 30.08.2016 um 21:10 schrieb Jiří Techet: [...]
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly.
One more time, seems I wasn't clear enough yet: I'm *not* suggesting that we create generic code inside Geany that handles any kind of AST. What I suggest is that plugins that use AST internally (or not) pass tag-like information to Geany, extracted and flattened from its internal AST (or whatever it uses). Then Geany and other plugins can use that information for their own purposes.
We do need more than just "tags-like information" though, for example, what's Geany gonna give for completions here (C++):
auto bar = foo<X>()->...
Or even on the next line:
bar.blah = baz... + 3
In the first case, it needs to know the type of not only `foo`'s return value, but it needs to know the type of `foo<X>`'s return type (could depend on `X`'s type and/or lots of other stuff).
For the second line, a super-smart auto-complete might only give suggestions for a `baz` that has a type that has an `::operator+` member that works with ints (or `baz` is a primitive numeric type), what's more, it could also filter the suggestions down so that it only shows identifiers (in any accessible scope) which start with `baz` and when the global or member `::operator+` is applied to it and 3, and the result of that which is also compatible with the `blah` member of `bar` (which is the type of the return of `foo<x>()` and never even mentioned locally).
As a further example, if we ever upgrade Geany's auto-complete list to be more than Scintilla's default (to something similar the one in GtkSourceView IIRC), which say for example showed the doc-comments for a given symbol (something libclang provides in/along with the AST, and is also accessible in Python AST as well, IIRC), not only now does TagManager have to store all that comment text (redundantly, or else call into the ft-plugin to get it on demand), but it has to pick-out the _exact_ right completion for the comment to be correct, based on all kinds of factors, such as the language's scope lookup rules (even inside of `if` blocks, which it also now has to store in the Tags array or recover from the source code some how), but also based on templated types once interpreted (TM would need to store some kind of decypherable mangled names after templates are expanded), and overloads (which it would need to determine given the given set of overloads, based on argument types which compile (and also SFINAE and all that jazz), and inheritance rules (TM would need to keep track of polymorphic, potentially multiply-inherited relationships), etc.
Basically, TM would have to have a semantic analyzer with the strength of that of a real compiler, for each language an ft-plugin might implement. That is what's needed for real Intellisense-like features and I believe what some of the responders are getting at. The same kind of smarts would be needed to also implement calltips or goto def/decl, though perhaps with slightly different logic.
What's more, for the specific libclang case, since it's meant to be used as a support library for IDEs (and other tools), it already provides a function like `completeAt(where, ...)` for auto-completion that when called provides a list of AST nodes referring to the valid completions at that point. It would be horrible to have to re-implement that function independent of the AST in TM/Geany, instead of just passing Geany a list of strings or TM tags to show in the popup list.
Cheers, Matthew Brush
On Tue, Aug 30, 2016 at 11:29 PM, Thomas Martitz kugel@rockbox.org wrote:
Am 30.08.2016 um 21:10 schrieb Jiří Techet:
Geany would then merge the tags, perhaps giving the pluginones more weight, and store it in TM. I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time.I think it would be a disaster performance-wise. The number of AST nodes can be easily 100x more than the amount of tags we have from ctags (we get a single tag for a function now and AST will contain complete tree for the function body) so just this might cost 100x more. In addition all the necessary copies to TM internal representation, having to maintain the tree structure (in TM we use GPtrArrays for everything which are very efficient and during tag merge we try to eliminate even pointer dereferences because those start getting expensive when managing many tags) etc.
Let's not outright reject possible solutions based on performance assumptions and guestimations. Performance can be evaluated on an actual implementation. Until then it's simply an invalid argument for this discussion. But FWIW, I don't think performance is the driving aspect.
No, performance is a very valid point. Tag updates don't happen in a background thread in Geany but rather on the main thread (and changing this would require lots of modifications as neither ctags, nor TM nor Geany are thread-safe) and all updates have to happen in a really short time period - you cannot make the GUI freeze while the user is typing so you have 100ms at most without any noticeable delay.
What's needed from the AST is tags (as you and I mentioned elsewhere, AST is the complete code representation, so much more than what's required currently). Those can be extracted in one pass. I don't see that tags need to be converted back to the original AST.
As Matthew said, tags are insufficient for good autocompletion (e.g. consider dynamic languages like Python where you have to infer variable type from the right-hand side of an assignment and based on that generate an autocompletion list).
For any successful plugin operation, the AST has to be generated (and probably re-generated regularly) and traversed at least once. Creating tags in a traversal that's happening done anyway probably isn't even going to add much overhead, if any.
But I assume AST is created in a background thread - as I said, all TM code runs on the main thread and everything has to happen fast enough between keystrokes so users don't experience any delays while typing.
As you say, we use a GPtrArray of tags because it's very efficient for sorting and merging. I think any ft-plugin will also have to sort (at least) for showing auto-completion and symbols tree - if it shows them itself. So it may even have to create such array/lists that TM uses anyway (you can't sort AST directly). So it might as well pass them to TM for sorting.
Whatever you do with tiny lists (say 1000 items) of tags which appear as a result of autocompletion doesn't matter. I'm talking about workspace array of tags which may contain all tags from a project with hundreds of thousands or millions of tags and which have to be updated as the user types and these updates have to be handled in a very efficient way.
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly.
One more time, seems I wasn't clear enough yet: I'm *not* suggesting that we create generic code inside Geany that handles any kind of AST. What I suggest is that plugins that use AST internally (or not) pass tag-like information to Geany, extracted and flattened from its internal AST (or whatever it uses). Then Geany and other plugins can use that information for their own purposes.
Again, for smart autocompletion you will need things from AST.
The biggest problem of current scope completion in Geany is the lack of information about local variables which we might get with the new cxx ctags parser and add them to TM. We'll get better results then but still AST-based autocompletion will be able to do smarter things.
Cheers,
Jiri
Am 31.08.2016 um 13:03 schrieb Jiří Techet:
On Tue, Aug 30, 2016 at 11:29 PM, Thomas Martitz <kugel@rockbox.org mailto:kugel@rockbox.org> wrote:
Am 30.08.2016 um 21:10 schrieb Jiří Techet: Geany would then merge the tags, perhaps giving the plugin ones more weight, and store it in TM. I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time. I think it would be a disaster performance-wise. The number of AST nodes can be easily 100x more than the amount of tags we have from ctags (we get a single tag for a function now and AST will contain complete tree for the function body) so just this might cost 100x more. In addition all the necessary copies to TM internal representation, having to maintain the tree structure (in TM we use GPtrArrays for everything which are very efficient and during tag merge we try to eliminate even pointer dereferences because those start getting expensive when managing many tags) etc. Let's not outright reject possible solutions based on performance assumptions and guestimations. Performance can be evaluated on an actual implementation. Until then it's simply an invalid argument for this discussion. But FWIW, I don't think performance is the driving aspect.No, performance is a very valid point. Tag updates don't happen in a background thread in Geany but rather on the main thread (and changing this would require lots of modifications as neither ctags, nor TM nor Geany are thread-safe) and all updates have to happen in a really short time period - you cannot make the GUI freeze while the user is typing so you have 100ms at most without any noticeable delay.
I'm saying we can't evaluate performance at this time, because no implementation is available. So saying now "uhm I fear it might be too slow my guess is that the other one is 100x faster" is just a wild assumption that doesn't help except spraying FUD. And outright rejecting a proposal based on such assumptions is invalid.
Please lets evaluate solutions, implement them, and then have an eye on performance.
What's needed from the AST is tags (as you and I mentioned elsewhere, AST is the complete code representation, so much more than what's required currently). Those can be extracted in one pass. I don't see that tags need to be converted back to the original AST.As Matthew said, tags are insufficient for good autocompletion (e.g. consider dynamic languages like Python where you have to infer variable type from the right-hand side of an assignment and based on that generate an autocompletion list).
Tags with incomplete information are insufficient. One solution can be to have tags hold sufficient information.
FWIW, unless you actually compile the stuff (for C++), any source code analysis will be insufficient for some cases. I was under the impression that we do *not* want mandatory compilation, which drags in build system implications, for stuff like auto completion.
But if a smart ft-plugin does this I don't care. I only care about if/how Geany - and other non-ft-plugins - can use the data from such an analysis.
For any successful plugin operation, the AST has to be generated (and probably re-generated regularly) and traversed at least once. Creating tags in a traversal that's happening done anyway probably isn't even going to add much overhead, if any.But I assume AST is created in a background thread - as I said, all TM code runs on the main thread and everything has to happen fast enough between keystrokes so users don't experience any delays while typing.
I don't care how the ft-plugin that provides tags to Geany generates it's internal AST. It can surely use a background thread. This is quite unrelated.
As you say, we use a GPtrArray of tags because it's very efficient for sorting and merging. I think any ft-plugin will also have to sort (at least) for showing auto-completion and symbols tree - if it shows them itself. So it may even have to create such array/lists that TM uses anyway (you can't sort AST directly). So it might as well pass them to TM for sorting.Whatever you do with tiny lists (say 1000 items) of tags which appear as a result of autocompletion doesn't matter. I'm talking about workspace array of tags which may contain all tags from a project with hundreds of thousands or millions of tags and which have to be updated as the user types and these updates have to be handled in a very efficient way.
Wait, a AST comprising a million nodes doesn't result in a million autocompletion candidates. Anyway, how does the number relate to our discussion?
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly. One more time, seems I wasn't clear enough yet: I'm *not* suggesting that we create generic code inside Geany that handles any kind of AST. What I suggest is that plugins that use AST internally (or not) pass tag-like information to Geany, extracted and flattened from its internal AST (or whatever it uses). Then Geany and other plugins can use that information for their own purposes.Again, for smart autocompletion you will need things from AST.
The biggest problem of current scope completion in Geany is the lack of information about local variables which we might get with the new cxx ctags parser and add them to TM. We'll get better results then but still AST-based autocompletion will be able to do smarter things.
Again, using AST is fine, within the ft-plugin. Use whatever you need from the AST for smart completion, and pass it in a generic format to Geany. This way Geany *will use* AST-based information just fine.
Best regards.
Le 31/08/2016 à 13:22, Thomas Martitz a écrit :
Am 31.08.2016 um 13:03 schrieb Jiří Techet:
[…]
No, performance is a very valid point. Tag updates don't happen in a background thread in Geany but rather on the main thread (and changing this would require lots of modifications as neither ctags, nor TM nor Geany are thread-safe) and all updates have to happen in a really short time period - you cannot make the GUI freeze while the user is typing so you have 100ms at most without any noticeable delay.
I'm saying we can't evaluate performance at this time, because no implementation is available. So saying now "uhm I fear it might be too slow my guess is that the other one is 100x faster" is just a wild assumption that doesn't help except spraying FUD. And outright rejecting a proposal based on such assumptions is invalid.
I wouldn't dismiss Jiří's performances remarks that easily :) Remember he spent a fair amount of time optimizing it, and has had experience with huge amount of tags with his project-organizer plugin on large projects (Linux kernel, anyone?). And IIUC, he realized that with all the tags *current* parsers generate (e.g. no local variables, etc.) TM started to struggle too much for it to be usable.
Sure, we need numbers, but I'm afraid we kinda already have some idea of what they would be.
Yes, maybe it'd be possible to improve the situation even further, but it might be a lot of work for very little gain. I would guess if a e.g. libclang works with real-size projects is because it has a lot of highly specific optimizations resulting of a lot of work on their side. And why I'm not sure it's even so useful for TM anyway, is that both Lex and Matthew showed some semantic examples that are both very subtle to deal with, and highly specific to some languages (here, C++) -- I love Lex's ADL example, C++ can seem just crazy :)
Regards, Colomban
I love Lex's ADL example, C++ can seem just crazy :)
Crazy like a Fox, makes the compiler near impossible, but makes lots of stuff "just work" :)
So a much simpler example then (using C syntax to explain since we all know it, but could be any language):
int a; { first_piece_of_code_using_a... float a; second_piece_of_code_using_a .... }
For C/C++ the first piece of code will see `a` as an int and the second piece of code will see `a` as a float, but there are languages (Julia for one) where both pieces of code will see `a` as a float. Again you need language specific knowledge to lookup `a` in the first piece of code.
Cheers Lex
Regards, Colomban _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 31.08.2016 um 15:31 schrieb Lex Trotman:
I love Lex's ADL example, C++ can seem just crazy :)
Crazy like a Fox, makes the compiler near impossible, but makes lots of stuff "just work" :)
So a much simpler example then (using C syntax to explain since we all know it, but could be any language):
int a; { first_piece_of_code_using_a... float a; second_piece_of_code_using_a .... }
For C/C++ the first piece of code will see `a` as an int and the second piece of code will see `a` as a float, but there are languages (Julia for one) where both pieces of code will see `a` as a float. Again you need language specific knowledge to lookup `a` in the first piece of code.
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Best regards.
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Geany must ask the plugin for the answer for each element of functionality the plugin provides. This means that those elements have indeed simply moved "Geany's deficiencies to the plugin space" as you put it. This is good and necessary.
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
The only place all the symbols should be returned is for the display in the symbol tree, and for that use I'll defer to Colomban's suggestion to provide TMTag structures, not for the plugin to access TM.
Best regards.
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
Geany must ask the plugin for the answer for each element of functionality the plugin provides. This means that those elements have indeed simply moved "Geany's deficiencies to the plugin space" as you put it. This is good and necessary.
To me this is not good because it makes it impossible to use the information for other use cases in Geany or other plugins.
For example, I want to keep my quickswitch plugin that allows me to jump to arbitrary tags, even if a ft-plugin is doing all the hard work to parse them.
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Best regards
On 2016-08-31 10:08 PM, Thomas Martitz wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
Let's use the simplest C example:
gint x = g_...
Please describe the algorithm (just in prose) that would give only the valid completions (ie. would compile in a C compiler) for `g_...` without providing any impossible completions, and without knowing anything about C/C++/Obj-C.
[...]
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
There's no reason an API couldn't be provided to query such information even from ft-plugins, for ex.
void query_symbols(criteria, out_tags_list);
Geany doesn't have to know every tag in order to call into ft-plugins for such query results, and can even fallback to TM/ctags if no plugins support this feature.
Cheers, Matthew Brush
On 2016-08-31 10:47 PM, Matthew Brush wrote:
On 2016-08-31 10:08 PM, Thomas Martitz wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
Let's use the simplest C example:
gint x = g_...Please describe the algorithm (just in prose) that would give only the valid completions (ie. would compile in a C compiler) for `g_...` without providing any impossible completions, and without knowing anything about C/C++/Obj-C.
To make it more interesting, you can assume that the full C/C++/Obj-C AST is somehow represented with full fidelity in a TM tags array, as a given.
Cheers, Matthew Brush
Am 01.09.2016 um 07:47 schrieb Matthew Brush:
On 2016-08-31 10:08 PM, Thomas Martitz wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
Let's use the simplest C example:
gint x = g_...Please describe the algorithm (just in prose) that would give only the valid completions (ie. would compile in a C compiler) for `g_...` without providing any impossible completions, and without knowing anything about C/C++/Obj-C.
Do you suggest the completion list would only contain function that return gint?
This is not what I would want. I can remember function names but not necessarily their exact return value. It's possible (even likely) that I got gint wrong and correct it to glong after auto completion, when the call tip for the completed function tells me that glong. It'd be annoying if auto-completion can't find the function I want just because I misremembered the return type and the thing tries to be too smart.
So, as for the algorithm, I'd really stick to prefix matching, as done currently. Everything else assumes that I coded everything correctly which is not always the case. I don't want to be auto completion to be too smart and hide wanted functions.
As Colomban and Jiri also mentioned it: auto-completion is also a useful tool for correcting typos and other programming errors.
[...]
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
There's no reason an API couldn't be provided to query such information even from ft-plugins, for ex.
void query_symbols(criteria, out_tags_list);
Geany doesn't have to know every tag in order to call into ft-plugins for such query results, and can even fallback to TM/ctags if no plugins support this feature.
Okay. Might as well pass the tags to Geany in the first place but it's important to make them available at all.
Best regards.
On 2016-09-01 01:55 AM, Thomas Martitz wrote:
Am 01.09.2016 um 07:47 schrieb Matthew Brush:
On 2016-08-31 10:08 PM, Thomas Martitz wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
Let's use the simplest C example:
gint x = g_...Please describe the algorithm (just in prose) that would give only the valid completions (ie. would compile in a C compiler) for `g_...` without providing any impossible completions, and without knowing anything about C/C++/Obj-C.
Do you suggest the completion list would only contain function that return gint?
Could be that, or it could be as simple as not listing things which aren't in scope or visible (say an #include is missing). What I was getting at is that unless you describe Geany's current algorithm, you'll almost surely start describing C/C++/Obj-specific semantics encoded inside Geany.
[...]
So, as for the algorithm, I'd really stick to prefix matching, as done currently. Everything else assumes that I coded everything correctly which is not always the case. I don't want to be auto completion to be too smart and hide wanted functions.
Ok, I see where the problem is. You seem to only want the current functionality while I want to enhance it to be more in-line with the type of smarts you get in real IDEs that have deep knowledge of your code and programming language. So if this is the case you're fundamentally not going to like any features ft-plugins provide, and I suggest that you just not use them. The point of adding all this in plugins is specifically so you don't have to :)
Cheers, Matthew Brush
On 1 September 2016 at 15:08, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what is the name of std::vector<int> and std::vector<A> so it can be looked up?
Geany must ask the plugin for the answer for each element of functionality the plugin provides. This means that those elements have indeed simply moved "Geany's deficiencies to the plugin space" as you put it. This is good and necessary.
To me this is not good because it makes it impossible to use the information for other use cases in Geany or other plugins.
For example, I want to keep my quickswitch plugin that allows me to jump to arbitrary tags, even if a ft-plugin is doing all the hard work to parse them.
Good point, the FT-Plugin API needs functions for goto declaration and goto definition. Unlike TM, the plugin can take language and scope rules into account when it answers, so you get the right declaration, continuing the theme above, where would you go to find the declaration of std::vector<int> ?
Certainly not the line:
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that `std::vector<int>` exists in the code.
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Only if your query plugin queries the FT plugin.
Best regards
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 01.09.2016 um 08:05 schrieb Lex Trotman:
On 1 September 2016 at 15:08, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 17:26 schrieb Lex Trotman:
I think we all agree that help of language-specific plugins is desired/required. No need to restate "we need language specific support" all the time.
We just disagree on how the language-specific knowledge is transported to Geany, other plugins and the user.
Well, I read your previous comments such as "But *just* the data, don't offload very specific use cases onto them" (and to some extent this one too) as saying that all the plugin has to do is return some data (tags) and Geany will take it from there and all I was doing was making sure it was clear that its unlikely that Geany will be able to "take it from there" in any meaningful way, based purely on data, without filetype specific knowledge inside Geany.
Here we disagree. I think it is possible to extent TM/Geany sufficiently to encode this information in generic ways. As an example, template instantiation yields different types. TM could tread them as different types. Any variable will be of one of the types, so calltips would just work.
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what is the name of std::vector<int> and std::vector<A> so it can be looked up?
vector<int> and vector<A>, with scope being std (I think trailing scope delimiters are not saved in the scope string of TMTag, just intermediate ones).
Geany must ask the plugin for the answer for each element of functionality the plugin provides. This means that those elements have indeed simply moved "Geany's deficiencies to the plugin space" as you put it. This is good and necessary.
To me this is not good because it makes it impossible to use the information for other use cases in Geany or other plugins.
For example, I want to keep my quickswitch plugin that allows me to jump to arbitrary tags, even if a ft-plugin is doing all the hard work to parse them.
Good point, the FT-Plugin API needs functions for goto declaration and goto definition. Unlike TM, the plugin can take language and scope rules into account when it answers, so you get the right declaration, continuing the theme above, where would you go to find the declaration of std::vector<int> ?
Certainly not the line:
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that `std::vector<int>` exists in the code.
Ideally the header which defines std::vector<A> would be found (and maybe opened) right? Currently this isn't possible with TMTag (as var_type can only contain either vector<A> or vector<int>) but it shouldn't be too hard to make it work. Like adding a generic_type field which is set for each template instance (containing vector<A>), then instead of looking up the decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have to provided by the ft-plugin too (maybe there needs to be a new TMTagType for vector<A> too).
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Only if your query plugin queries the FT plugin.
So that means
1) My plugin contains code specifically for each known ft-plugin (i.e. not the plugin API) 2) There needs to be a shared library which my plugin can use to call into the other plugin 3) My plugin needs to know if and when to ask a ft-plugin
This is not acceptable. Or are you saying Geany could provide a wrapper for this as Matthew suggested? That might be workable.
Best regards.
[...]
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what is the name of std::vector<int> and std::vector<A> so it can be looked up?
vector<int> and vector<A>, with scope being std (I think trailing scope delimiters are not saved in the scope string of TMTag, just intermediate ones).
Sorry Thomas, I was a little mean and laid a trap.
std::vector has a second parameter, which has a default value so its legal to omit it, but the type you will get from an accurate parser to store in TM will be the complete type which will be something like std::vector<int, std::allocator<int>> not just plain std::vector<int>. This is the type you see printed in g++ and clang++ error messages for template expansions, very messy.
So the way TM does a pure textual comparison of the prefix in the source code `std::vector<int>` simply won't work to find `std::vector<int,std::allocator<int>>`
It also won't work if there are spaces as I showed in a previous post `std::vector< int >` is perfectly legal C++ but won't match `std::vector<int>`. And before you ask, yes lots of code has the spaces so that nested angle bracket closes (as in my ...allocator<int>> above) look like > > not the >> operator. Being able to omit the space and not have it seen as a shift right operator is only a recent addition to the standard.
And for template functions declared as `template<class T>T& f(T& t){ return t<<1; }` they are used as simply `int a = f(1);` not an angle bracket in sight, and a textual comparison to find the declaration for a calltip isn't going to work because no such explicit declaration exists. So how can TM answer whats legal here `f(std::cout).` (answer all of std::ostream) versus here `f(1).` (answer nothing).
[...]
Good point, the FT-Plugin API needs functions for goto declaration and goto definition. Unlike TM, the plugin can take language and scope rules into account when it answers, so you get the right declaration, continuing the theme above, where would you go to find the declaration of std::vector<int> ?
Certainly not the line:
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that `std::vector<int>` exists in the code.
Ideally the header which defines std::vector<A> would be found (and maybe opened) right? Currently this isn't possible with TMTag (as var_type can only contain either vector<A> or vector<int>) but it shouldn't be too hard to make it work.
But now you are encoding language specific semantics into TM, and as I noted above its more complicated than just that example.
Please don't get the idea from simple examples posted here that they cover even a fraction of the craziness of languages (and not just C++ either, its simply the one I'm most familiar with).
Like adding a generic_type field which is set for each
template instance (containing vector<A>), then instead of looking up the decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have to provided by the ft-plugin too (maybe there needs to be a new TMTagType for vector<A> too).
And don't forget that when trying to look up `std::vector<int>::value_type` to see whats next, or to find the declaration, either Geany or TM has to parse it and break it into the elements `std`, `vector<int>` and `value_type`, ie has to know the language syntax for template specialisations and namespaces, `vector<int>` is not a legal name. And as I said before, that syntax varies between languages, so again you are encoding language specifics into Geany or TM.
And in a final real piece of evil C++ for the day, variadic templates with variable numbers of parameters (like C variadic functions with ...).
std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(
ok, lets lookup the return type of std::make_tuple http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats simple-ish and the type of std::get<1> http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm
And yep its all completely legal and idiomatic C++ and totally static typed so should be no problem analysing it.
Sorry Colomban you didn't want see half the craziness of C++ I know :)
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Only if your query plugin queries the FT plugin.
So that means
- My plugin contains code specifically for each known ft-plugin (i.e. not
the plugin API)
No, but the implementation inside Geany of the query interface that you have added in your PR may have to know to call the FT-plugin to answer the query, but the plugin using the query interface doesn't need to know about that.
- There needs to be a shared library which my plugin can use to call into
the other plugin
I'm not sure why you would need a third dll even if you were having to call the FT-plugin directly, you probably have to be passed its g_module though. But you shouldn't have to access the FT-plugin directly unless you are doing something Geany doesn't support.
- My plugin needs to know if and when to ask a ft-plugin
As I said unless you are asking a question Geany doesn't know how to answer, then you shouldn't need to.
This is not acceptable.
Be aware that as you say that, the alternative is for you to implement all those language specifics I mentioned above in your plugin itself.
Or are you saying Geany could provide a wrapper for this as Matthew suggested? That might be workable.
Yes, exactly. As Matthew said in a previous post, for each feature where a FT-plugin is involved the function inside Geany that provides that feature is going to have to have an if near the front that makes the decision if it uses the plugin or does it itself. So if your plugin uses that function you get the benefits of both implementations.
Cheers Lex
Best regards.
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 01.09.2016 um 15:00 schrieb Lex Trotman:
[...]
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what is the name of std::vector<int> and std::vector<A> so it can be looked up?
vector<int> and vector<A>, with scope being std (I think trailing scope delimiters are not saved in the scope string of TMTag, just intermediate ones).
Sorry Thomas, I was a little mean and laid a trap.
No problem. The devil is always in the details and we can only define requirements when looking and complicated cases.
std::vector has a second parameter, which has a default value so its legal to omit it, but the type you will get from an accurate parser to store in TM will be the complete type which will be something like std::vector<int, std::allocator<int>> not just plain std::vector<int>. This is the type you see printed in g++ and clang++ error messages for template expansions, very messy.
So the way TM does a pure textual comparison of the prefix in the source code `std::vector<int>` simply won't work to find `std::vector<int,std::allocator<int>>`
It also won't work if there are spaces as I showed in a previous post `std::vector< int >` is perfectly legal C++ but won't match `std::vector<int>`. And before you ask, yes lots of code has the spaces so that nested angle bracket closes (as in my ...allocator<int>> above) look like > > not the >> operator. Being able to omit the space and not have it seen as a shift right operator is only a recent addition to the standard.
OK, so we'd have to have a canonical representation of types (how this looks like is defined by the ft-plugin, e.g. whitespaces stripped, default parameters expanded) to compare against, and a method to find that tag when looking up std::vector< int >. It seems be complicated indeed, but doable. Most of the complexity will be in the ft-plugin providing canonicalized, tag-like data.
And for template functions declared as `template<class T>T& f(T& t){ return t<<1; }` they are used as simply `int a = f(1);` not an angle bracket in sight, and a textual comparison to find the declaration for a calltip isn't going to work because no such explicit declaration exists. So how can TM answer whats legal here `f(std::cout).` (answer all of std::ostream) versus here `f(1).` (answer nothing).
I would say that a calltip for a templated function should show the generic function since you don't know what the user is going to type, regardless of the design (except perhaps if the specialized function can be inferred from the scope). Which specialized function is to be called changes as the user types. As I said in the other mail to Matthew, trying to infer from the lhs of the assignment is dangerous and impractical too (I would not want this behavior), especially if the lhs is determined only by the rhs (auto x = foo(1)).
Perhaps you can show specialized calltips once user has entered a few parameters but that's still guessing. I don't think this is a good idea.
But plain textual comparison against the function name seems to work best here to me.
[...]
Good point, the FT-Plugin API needs functions for goto declaration and goto definition. Unlike TM, the plugin can take language and scope rules into account when it answers, so you get the right declaration, continuing the theme above, where would you go to find the declaration of std::vector<int> ?
Certainly not the line:
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that `std::vector<int>` exists in the code.
Ideally the header which defines std::vector<A> would be found (and maybe opened) right? Currently this isn't possible with TMTag (as var_type can only contain either vector<A> or vector<int>) but it shouldn't be too hard to make it work.
But now you are encoding language specific semantics into TM, and as I noted above its more complicated than just that example.
Generic ways to deal with language specific properties isn't quite the same. Also, this doesn't apply to just C++, other languages have complex meta programming / generic types too (Vala, Java, C#, etc.).
And even then, I'm totally fine with TM knowing language specifics. TM can hold every information needed to make smart language specific features work, preferably in generic ways. That doesn't mean TM has to semantically understand the language's source code, if the information is provided by external parsers. In fact, it doesn't have to understand source code itself at all.
There may be a point we want to limit the smartness anyway, if requires very complex solutions for little gain, extremely rare use cases or they it can become annoying to the user (e.g. if it requires to much setup).
Please don't get the idea from simple examples posted here that they cover even a fraction of the craziness of languages (and not just C++ either, its simply the one I'm most familiar with).
Like adding a generic_type field which is set for each
template instance (containing vector<A>), then instead of looking up the decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have to provided by the ft-plugin too (maybe there needs to be a new TMTagType for vector<A> too).
And don't forget that when trying to look up `std::vector<int>::value_type` to see whats next, or to find the declaration, either Geany or TM has to parse it and break it into the elements `std`, `vector<int>` and `value_type`, ie has to know the language syntax for template specialisations and namespaces, `vector<int>` is not a legal name. And as I said before, that syntax varies between languages, so again you are encoding language specifics into Geany or TM.
I don't understand that. TM doesn't need to know that. It just has some strings contained in tags that are matched against other strings. TM doesn't break A::B::X, the parser does (currently this is split into A::B and X). TM already has the concept of scope for that. It works as of now, without knowing langauge specific syntax.
And in a final real piece of evil C++ for the day, variadic templates with variable numbers of parameters (like C variadic functions with ...).
std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(
ok, lets lookup the return type of std::make_tuple http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats simple-ish and the type of std::get<1> http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm
And yep its all completely legal and idiomatic C++ and totally static typed so should be no problem analysing it.
Sorry Colomban you didn't want see half the craziness of C++ I know :)
I think we the above suggestions this should work. Sure it must be parsed by a ft-plugin.
You're heavy on templates which require compilation. I'm interested to see how you want realize this (passing CFLAGS etc.) in a user friendly way on a per-ft-plugin basis.
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Only if your query plugin queries the FT plugin.
So that means
- My plugin contains code specifically for each known ft-plugin (i.e. not
the plugin API)
No, but the implementation inside Geany of the query interface that you have added in your PR may have to know to call the FT-plugin to answer the query, but the plugin using the query interface doesn't need to know about that.
Okay. That's at least one more good reason to have the query interface :-)
Best regards
[...]>>
std::vector has a second parameter, which has a default value so its legal to omit it, but the type you will get from an accurate parser to store in TM will be the complete type which will be something like std::vector<int, std::allocator<int>> not just plain std::vector<int>. This is the type you see printed in g++ and clang++ error messages for template expansions, very messy.
So the way TM does a pure textual comparison of the prefix in the source code `std::vector<int>` simply won't work to find `std::vector<int,std::allocator<int>>`
It also won't work if there are spaces as I showed in a previous post `std::vector< int >` is perfectly legal C++ but won't match `std::vector<int>`. And before you ask, yes lots of code has the spaces so that nested angle bracket closes (as in my ...allocator<int>> above) look like > > not the >> operator. Being able to omit the space and not have it seen as a shift right operator is only a recent addition to the standard.
OK, so we'd have to have a canonical representation of types (how this looks like is defined by the ft-plugin, e.g. whitespaces stripped, default parameters expanded) to compare against, and a method to find that tag when looking up std::vector< int >. It seems be complicated indeed, but doable. Most of the complexity will be in the ft-plugin providing canonicalized, tag-like data.
There is no guarantee that a gcc based library (assuming RMS allows it one day) would use the same canonical form as libclang, in fact it probably deliberately would not, so its hard to swap them, so now you not only have language specific code in Geany, but FT-plugin specific code. And for other languages FT-plugins too.
And it still doesn't address the default template parameter issue above.
And for template functions declared as `template<class T>T& f(T& t){ return t<<1; }` they are used as simply `int a = f(1);` not an angle bracket in sight, and a textual comparison to find the declaration for a calltip isn't going to work because no such explicit declaration exists. So how can TM answer whats legal here `f(std::cout).` (answer all of std::ostream) versus here `f(1).` (answer nothing).
I would say that a calltip for a templated function should show the generic function since you don't know what the user is going to type, regardless of the design (except perhaps if the specialized function can be inferred from the scope). Which specialized function is to be called changes as the user types. As I said in the other mail to Matthew, trying to infer from the lhs of the assignment is dangerous and impractical too (I would not want this behavior), especially if the lhs is determined only by the rhs (auto x = foo(1)).
Yes, in C++ you can't infer from left to right, but there are languages where this happens, Haskel, O'Caml that I immediately think of. But my example was just to point out that the template parameter is inferred from the type of the function parameter, the user does not specify `f<int>(1)` sorry for the confusion.
And yes, as I showed in another post, accurate calltips can change as parameters are added.
But the last two examples above are not calltips, they are autocomplete, the function call is finished.
Both end in a dot accessing members of the return from the function. To get accurate type info it is necessary to first recognise that a template is needed for `f` (there are no <>s in the expressions) then recognise that its return type matches its parameter type, then you can suggest what members can come after the dot.
And just to note neither the return or the function parameter need to be exactly the template parameter, for example `template<class T> T& f(T* a)` will happily deduce a `T` from a `T*` and make the return type `T&`. And several other type expressions work.
Perhaps you can show specialized calltips once user has entered a few parameters but that's still guessing. I don't think this is a good idea.
But the FT-plugin may do this (at least IIUC libclang will) so its fine if Geany doesn't, but it has to ask the FT-plugin not use TM.
But plain textual comparison against the function name seems to work best here to me.
Depends what you are coding, as the car manufacturers say, "your mileage may vary".
[...]
Good point, the FT-Plugin API needs functions for goto declaration and goto definition. Unlike TM, the plugin can take language and scope rules into account when it answers, so you get the right declaration, continuing the theme above, where would you go to find the declaration of std::vector<int> ?
Certainly not the line:
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that `std::vector<int>` exists in the code.
Ideally the header which defines std::vector<A> would be found (and maybe opened) right? Currently this isn't possible with TMTag (as var_type can only contain either vector<A> or vector<int>) but it shouldn't be too hard to make it work.
But now you are encoding language specific semantics into TM, and as I noted above its more complicated than just that example.
Generic ways to deal with language specific properties isn't quite the same. Also, this doesn't apply to just C++, other languages have complex meta programming / generic types too (Vala, Java, C#, etc.).
And even then, I'm totally fine with TM knowing language specifics.
I keep saying Geany as well, not just TM, the language specifics to storing stuff is one thing, the language specifics for using that stored stuff is another more complex thing. And, if you have to ask the plugin for the more complex questions, there is no point of storing all the gory details in TM, because the plugin isn't going to use it.
TM can hold every information needed to make smart language specific features work, preferably in generic ways. That doesn't mean TM has to semantically understand the language's source code, if the information is provided by external parsers. In fact, it doesn't have to understand source code itself at all.
So as originally proposed, Geany has to ask the plugin first.
There may be a point we want to limit the smartness anyway, if requires very complex solutions for little gain, extremely rare use cases or they it can become annoying to the user (e.g. if it requires to much setup).
To emphasise I am not saying don't make TM and Geany smarter, but I am saying don't make a `c.c` like mess of language specific code in Geany or TM, and don't say "this is good enough for me so it should be good enough for you" and prevent the use of smarter plugins for those who want them. That means the plugin has to bypass the code in TM/Geany to bypass its restrictions.
Please don't get the idea from simple examples posted here that they cover even a fraction of the craziness of languages (and not just C++ either, its simply the one I'm most familiar with).
Like adding a generic_type field which is set for each
template instance (containing vector<A>), then instead of looking up the decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have to provided by the ft-plugin too (maybe there needs to be a new TMTagType for vector<A> too).
And don't forget that when trying to look up `std::vector<int>::value_type` to see whats next, or to find the declaration, either Geany or TM has to parse it and break it into the elements `std`, `vector<int>` and `value_type`, ie has to know the language syntax for template specialisations and namespaces, `vector<int>` is not a legal name. And as I said before, that syntax varies between languages, so again you are encoding language specifics into Geany or TM.
I don't understand that. TM doesn't need to know that. It just has some strings contained in tags that are matched against other strings. TM doesn't break A::B::X, the parser does (currently this is split into A::B and X). TM already has the concept of scope for that. It works as of now, without knowing langauge specific syntax.
ATM Geany does that breakup of the expression in the source code and asks TM for each element.
As I said above, if the plugin parser is doing it, the plugin parser will look it up in its accurate language specific symbol table as well, it won't use TM. So no need to store language specifics in TM if you are using such a library.
To go back to one of your very initial comments "don't want to move everything from TM/Geany to the plugin" my intention in all these examples is to show that an accurate plugin MUST bypass TM/Geany because they will always have limitations without effectively having the same code as the plugin, ie a compiler.
And in a final real piece of evil C++ for the day, variadic templates with variable numbers of parameters (like C variadic functions with ...).
std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(
ok, lets lookup the return type of std::make_tuple http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats simple-ish and the type of std::get<1> http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm
And yep its all completely legal and idiomatic C++ and totally static typed so should be no problem analysing it.
Sorry Colomban you didn't want see half the craziness of C++ I know :)
I think we the above suggestions this should work. Sure it must be parsed by a ft-plugin.
You're heavy on templates which require compilation. I'm interested to see how you want realize this (passing CFLAGS etc.) in a user friendly way on a per-ft-plugin basis.
Yes, to get the benefit of the plugins wisdom, the price is that it needs to know how the software is to be built (at least for libclang). But thats the plugins problem, not Geanys. Its a trade off. It might not be a worthwhile trade off for you, but don't disallow usage by others for whom the tradeoff is different. And as its a plugin you can decide to not use it.
Of course moving the problem to plugin space doesn't mean the plugin can't use Geany facilities where their capabilities fit the requirement. But we should not try to expand Geany to handle all types of functionality.
Like your TM query interface, the plugins should answer the questions like "whats the autocomplete here", "what calltips are relevant here" with a flat list of data relevant to the question.
My TM query interface wants to return all matching tags, including those found by ft-plugins. Can this be done?
Only if your query plugin queries the FT plugin.
So that means
- My plugin contains code specifically for each known ft-plugin (i.e.
not the plugin API)
No, but the implementation inside Geany of the query interface that you have added in your PR may have to know to call the FT-plugin to answer the query, but the plugin using the query interface doesn't need to know about that.
Okay. That's at least one more good reason to have the query interface :-)
I havn't looked in detail at your query interface, but the approach of providing general wrapper APIs for external use, instead of exposing the guts of Geany, I very much support. Remember waaaay back when we were doing the build system, I proposed a set of functions to expose it to the plugin interface in a generic manner, but they were vetoed by Nick.
Cheers Lex
Best regards
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
On Wed, Aug 31, 2016 at 1:22 PM, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 13:03 schrieb Jiří Techet:
On Tue, Aug 30, 2016 at 11:29 PM, Thomas Martitz <kugel@rockbox.org mailto:kugel@rockbox.org> wrote:
Am 30.08.2016 um 21:10 schrieb Jiří Techet: Geany would then merge the tags, perhaps giving the plugin ones more weight, and store it in TM. I think you underestimate how many tags we're talking here. The example libclang ft-plugin would have to re-walk the entire AST (which is absolutely massive, particularly for C++), convert it to TM tag structures, and then Geany/TM would have to perform some merging logic, would would be more complicated than now if it was to support C++ properly, every single re-parse. My intuition tells me that just won't be fast enough, Clang already jumps through hoops and uses tricks to just build its own AST in-time. I think it would be a disaster performance-wise. The number of AST nodes can be easily 100x more than the amount of tags we have from ctags (we get a single tag for a function now and AST will contain complete tree for the function body) so just this might cost 100x more. In addition all the necessary copies to TM internal representation, having to maintain the tree structure (in TM we use GPtrArrays for everything which are very efficient and during tag merge we try to eliminate even pointer dereferences because those start getting expensive when managing many tags) etc. Let's not outright reject possible solutions based on performance assumptions and guestimations. Performance can be evaluated on an actual implementation. Until then it's simply an invalid argument for this discussion. But FWIW, I don't think performance is the driving aspect.No, performance is a very valid point. Tag updates don't happen in a background thread in Geany but rather on the main thread (and changing this would require lots of modifications as neither ctags, nor TM nor Geany are thread-safe) and all updates have to happen in a really short time period - you cannot make the GUI freeze while the user is typing so you have 100ms at most without any noticeable delay.
I'm saying we can't evaluate performance at this time, because no implementation is available. So saying now "uhm I fear it might be too slow my guess is that the other one is 100x faster" is just a wild assumption that doesn't help except spraying FUD. And outright rejecting a proposal based on such assumptions is invalid.
100x more nodes of AST than the amount of tags we have isn't any wild assumption - have a look at the AST picture here:
https://en.wikipedia.org/wiki/Abstract_syntax_tree
and the primitive code below to which it corresponds. The tree has 21 nodes. It's easy to imagine source files with 5x more code per function and you are at the numbers I'm giving you - we generate 1 tag per function and the corresponding AST you'll have to walk is 100x bigger.
Please lets evaluate solutions, implement them, and then have an eye on performance.
I think I have evaluated this solution and it doesn't seem worth the effort. Not only because the performance but also I don't know how to store all the necessary information for any kind of language into TM so code completion works for any language so that it's compatible with ctags and any kind of AST. If you do, please tell us but not the way "One solution can be to have tags hold sufficient information" because it doesn't tell anything.
What's needed from the AST is tags (as you and I mentioned elsewhere, AST is the complete code representation, so much more than what's required currently). Those can be extracted in one pass. I don't see that tags need to be converted back to the original AST.As Matthew said, tags are insufficient for good autocompletion (e.g. consider dynamic languages like Python where you have to infer variable type from the right-hand side of an assignment and based on that generate an autocompletion list).
Tags with incomplete information are insufficient. One solution can be to have tags hold sufficient information.
FWIW, unless you actually compile the stuff (for C++), any source code analysis will be insufficient for some cases. I was under the impression that we do *not* want mandatory compilation, which drags in build system implications, for stuff like auto completion.
libclang basically compiles it - it performs complete syntax analysis the same way that would happen during compilation. It just skips code generation.
But if a smart ft-plugin does this I don't care. I only care about if/how Geany - and other non-ft-plugins - can use the data from such an analysis.
For any successful plugin operation, the AST has to be generated (and probably re-generated regularly) and traversed at least once. Creating tags in a traversal that's happening done anyway probably isn't even going to add much overhead, if any.But I assume AST is created in a background thread - as I said, all TM code runs on the main thread and everything has to happen fast enough between keystrokes so users don't experience any delays while typing.
I don't care how the ft-plugin that provides tags to Geany generates it's internal AST. It can surely use a background thread. This is quite unrelated.
I was reacting to you where you said that AST has to be regenerated regularly and that time spent on updating TM will be smaller. Yes, it will be smaller, but it will have to be performed on the main thread instead of in the background so the performance will be much more critical.
As you say, we use a GPtrArray of tags because it's very efficient for sorting and merging. I think any ft-plugin will also have to sort (at least) for showing auto-completion and symbols tree - if it shows them itself. So it may even have to create such array/lists that TM uses anyway (you can't sort AST directly). So it might as well pass them to TM for sorting.Whatever you do with tiny lists (say 1000 items) of tags which appear as a result of autocompletion doesn't matter. I'm talking about workspace array of tags which may contain all tags from a project with hundreds of thousands or millions of tags and which have to be updated as the user types and these updates have to be handled in a very efficient way.
Wait, a AST comprising a million nodes doesn't result in a million autocompletion candidates. Anyway, how does the number relate to our discussion?
It appears to me you don't really read what I'm writing - I'm talking about updating TM workspace tags which has to happen every time file is reparsed. As I said, autocompletion list population performance doesn't matter much because it will contain a lot less tags.
And by the way, projects with 1 000 000 tags will have more like 100 000 000 nodes in AST.
And even if we did this, I don't know how we could handle ASTs of different languages in a generic way because these will differ significantly. One more time, seems I wasn't clear enough yet: I'm *not* suggesting that we create generic code inside Geany that handles any kind of AST. What I suggest is that plugins that use AST internally (or not) pass tag-like information to Geany, extracted and flattened from its internal AST (or whatever it uses). Then Geany and other plugins can use that information for their own purposes.Again, for smart autocompletion you will need things from AST.
The biggest problem of current scope completion in Geany is the lack of information about local variables which we might get with the new cxx ctags parser and add them to TM. We'll get better results then but still AST-based autocompletion will be able to do smarter things.
Again, using AST is fine, within the ft-plugin. Use whatever you need from the AST for smart completion, and pass it in a generic format to Geany. This way Geany *will use* AST-based information just fine.
I'm ending the discussion here, don't expect more contributions on this topic from me. You don't listen to what others are saying and keep rejecting their arguments without giving any concrete proposals other than "Use whatever you need" and "and pass it in a generic format to Geany" which tell nothing. Continuing in this is a waste of time for me.
Jiri
On 2016-08-31 08:15 AM, Jiří Techet wrote:
On Wed, Aug 31, 2016 at 1:22 PM, Thomas Martitz kugel@rockbox.org wrote: [...]
Wait, a AST comprising a million nodes doesn't result in a million autocompletion candidates. Anyway, how does the number relate to our discussion?
It appears to me you don't really read what I'm writing - I'm talking about updating TM workspace tags which has to happen every time file is reparsed. As I said, autocompletion list population performance doesn't matter much because it will contain a lot less tags.
And by the way, projects with 1 000 000 tags will have more like 100 000 000 nodes in AST.
To give a sensational example, a minimal "hello world" C++ program:
#include <iostream> int main() { std::cout << "Hello World" << std::endl; return 0; }
CTags:
$ ctags -o - hello-world.cc | wc -l 1
Clang AST:
$ clang++ -Xclang -ast-dump -fno-diagnostics-color \ -fsyntax-only hello-world.cc | wc -l 37806
Obviously since C++ compilers operate on translation units, the latter includes all of what's brought in through includes. If I manually strip it down to just what humans see in the source file, it's around 15 lines, but the 37806 is what a libclang ft-plugin would be working with in the AST.
I couldn't speculate what a larger project like LLVM itself would parse to, but I suspect it would take a not-insignificant amount of time for an ft-plugin to even traverse the AST once (eg. to find all the nodes originating in a particular source file).
Cheers, Matthew Brush
On 29 August 2016 at 22:38, Thomas Martitz kugel@rockbox.org wrote:
Am 29.08.2016 um 14:23 schrieb Lex Trotman:
This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases. The point of this proposal is to change and add use-cases that are not currently possible with the current plugin API. But instead of each use-case generating its own piece of API and its own infrastructure, the point of the FT-plugins proposal is to provide a common infrastructure and approach for all filetype specific use-cases, those needed for currently suggested uses, indentation, clang based styling and symbols, and as framework for future use-cases we either havn't thought of, or havn't a concrete intention to add immediately.
I thought we agreed that plugin should simply provide tags to Geany/TM
This proposal is about many types of filetype specific functionality, not just tags. Tagmanager will not help in any way with indenting Haskell, or even C++.
4 of 5 of the proposed features are strictly tag-related. And Geany can do all of them already, it's just that the current implementation leaves things to be desired so there is the idea to let plugins improve upon them.
Well, 3 out of 6 but whos counting :)
Certainly 1) showing symbols in the symbol list, 2) autocomplete and 3) calltips are currently available to a degree in Geany. But highlighting, build commands and build result handling are not. But to be able to do 2) and 3) accurately needs more knowledge of each language semantics than is currently available in Geany or tagmanager.
I disagree with the proposed solution for those 4, because they are offloading logic on a per feature basis to plugins, only because Geany isn't capable at the moment. If Geany was capable, then there could be 1 solution for the 4 features and less complexity in each plugin (and we know the quality of plugins varies a lot so they should have little complexity as possible).
Encoding the knowledge of language semantics into Geany, for each language supported, is going to make autocomplete and calltip code look like c.c. Its not the way to go.
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.
Well, again you are encoding language semantics into Geany, for example for C++ that means autocompletion and calltips need to handle 1) local symbol scopes, 2) member functions being in the scope of the class, even when they are not 3) argument dependent lookup 4) template expansion lookup and 5) handling of template parameter based typing. These are hard, just ask the GCC and clang guys. And every user of Geany will have to pay the cost of the code they don't use, unless they use C++.
Then for a multidispatch language like Julia you need to handle overloading in an even more subtle way than C++ overloading.
And why re-implement these language specific subtle and difficult features in Geany when more and more languages are providing libclang like libraries to do it for us, accurately and up to date with the language.
Cheers Lex
Best regards. _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 30.08.2016 um 03:56 schrieb Lex Trotman:
On 29 August 2016 at 22:38, Thomas Martitz kugel@rockbox.org wrote:
Am 29.08.2016 um 14:23 schrieb Lex Trotman:
This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases. The point of this proposal is to change and add use-cases that are not currently possible with the current plugin API. But instead of each use-case generating its own piece of API and its own infrastructure, the point of the FT-plugins proposal is to provide a common infrastructure and approach for all filetype specific use-cases, those needed for currently suggested uses, indentation, clang based styling and symbols, and as framework for future use-cases we either havn't thought of, or havn't a concrete intention to add immediately.
I thought we agreed that plugin should simply provide tags to Geany/TM
This proposal is about many types of filetype specific functionality, not just tags. Tagmanager will not help in any way with indenting Haskell, or even C++.
4 of 5 of the proposed features are strictly tag-related. And Geany can do all of them already, it's just that the current implementation leaves things to be desired so there is the idea to let plugins improve upon them.
Well, 3 out of 6 but whos counting :)
I count "Go To Declaration/Definition" as a tag issue. Where a symbol is stored, and whether it's a definition or declaration, is contained in tags.
I did not count the build/run support, diagnostics and refactoring because there was no specific propsal given.
Certainly 1) showing symbols in the symbol list, 2) autocomplete and 3) calltips are currently available to a degree in Geany. But highlighting, build commands and build result handling are not.
But to be able to do 2) and 3) accurately needs more knowledge of each language semantics than is currently available in Geany or tagmanager.
That's right. But it doesn't mean the features should be *entirely* moved into plugin space. TM should be improved to be able to hold sufficient information, and plugins should help by providing that data. But *just* the data, don't offload very specific use cases onto them, this will make us less flexible in the long run. If Geany has the data, we can implement new features inside Geany or non-ft-plugins. Otherwise we would have to modify every single ft-plugin for new features and exclude non-ft-plugins.
I disagree with the proposed solution for those 4, because they are offloading logic on a per feature basis to plugins, only because Geany isn't capable at the moment. If Geany was capable, then there could be 1 solution for the 4 features and less complexity in each plugin (and we know the quality of plugins varies a lot so they should have little complexity as possible).
Encoding the knowledge of language semantics into Geany, for each language supported, is going to make autocomplete and calltip code look like c.c. Its not the way to go.
Nobody suggested to encode specific language semantics into Geany. I'm suggesting TM should become more generic, i.e. TMTag should transport as much information as required (even if some of it can't be provided by ctags but only some plugins).
The solution I have in mind simply allows plugins to pass tags to Geany which they parsed with more advanced code. The tags itself would advanced too, to allow for the improvements current TM+ctags can't offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be improved based on the advanced tags.
Well, again you are encoding language semantics into Geany, for example for C++ that means autocompletion and calltips need to handle
- local symbol scopes, 2) member functions being in the scope of the
class, even when they are not 3) argument dependent lookup 4) template expansion lookup and 5) handling of template parameter based typing. These are hard, just ask the GCC and clang guys. And every user of Geany will have to pay the cost of the code they don't use, unless they use C++.
Then for a multidispatch language like Julia you need to handle overloading in an even more subtle way than C++ overloading.
And why re-implement these language specific subtle and difficult features in Geany when more and more languages are providing libclang like libraries to do it for us, accurately and up to date with the language.
Most of the issues don't apply to just C++ but so many other languages as well (e.g. Java). All of them can be improved more easily, not just C++, if Geany can offer a powerful framework for that. But the framework Matthew proposed is not powerful to me. It just evades the current problems simply by moving Geany's deficiencies to the plugin space.
Best regards
Le 30/08/2016 à 17:31, Thomas Martitz a écrit :
Am 30.08.2016 um 03:56 schrieb Lex Trotman:
[…]
Certainly 1) showing symbols in the symbol list, 2) autocomplete and 3) calltips are currently available to a degree in Geany. But highlighting, build commands and build result handling are not.
But to be able to do 2) and 3) accurately needs more knowledge of each language semantics than is currently available in Geany or tagmanager.
That's right. But it doesn't mean the features should be *entirely* moved into plugin space. TM should be improved to be able to hold sufficient information, and plugins should help by providing that data. But *just* the data, don't offload very specific use cases onto them, this will make us less flexible in the long run. If Geany has the data, we can implement new features inside Geany or non-ft-plugins. Otherwise we would have to modify every single ft-plugin for new features and exclude non-ft-plugins.
I don't think the real problem for the current state to really only be about what tags are generated. Sure, if tags for local variables were generated, scope completion would become more useful just like that indeed. And then, the current code should only be improved to properly use the tags to show completions.
But this doesn't take into account several language specific things, like:
* what is an identifier? (for looking up what to complete) * what is an argument list? (for showing calltips) * what are the scope semantics? * …
Sure, some of those can be implemented generically using TagManager capabilities with appropriate tags, just by improving the generic code. But that means scope semantics are always the same, or are pluggable, and same for the other details.
Yes, I'd love TagManager (and our use of it rather) to be as good as it can get, and it could get pretty far (like what I tried with that "visible in current scope" patch in the scoped calltip PR). It's not trivial (like requires to know all the imported namespaces and stuff), but it's doable if all languages can fit in the same jar [1]
But if a language has totally different semantics than, say, C++, we'd be pretty much screwed no matter how good the C++ semantics were.
Also, I think Matthew pointed out that in any case, to get perfect completions we'd need prefect tags, and that we'd have to check how fast it is with virtually an infinite set of tags (Jiří?). But that's indeed not a concern to have now.
Anyway, my point is that I totally agree that we should try to get TM completions to the best they can be, but that no matter how good it is won't ever be as good (well, at least not better) than completely specific code. So I think Matthew's goal of allowing to *replace* code deciding what to complete makes sense.
Regards, Colomban
[1] I don't know the English expression, so I built one that sounded nice :)
[...]
But to be able to do 2) and 3) accurately needs more knowledge of each language semantics than is currently available in Geany or tagmanager.
That's right. But it doesn't mean the features should be *entirely* moved into plugin space. TM should be improved to be able to hold sufficient information, and plugins should help by providing that data. But *just* the data, don't offload very specific use cases onto them, this will make us less flexible in the long run. If Geany has the data, we can implement new features inside Geany or non-ft-plugins. Otherwise we would have to modify every single ft-plugin for new features and exclude non-ft-plugins.
The problem isn't having the data, its what you do with the data. To repeat two C++ examples I posted on IRC:
namespace foo { struct A { int a=1; int b=2; }; void f(A& anA, int inc){ anA.a+= inc; } };
foo::A anA; std::vectorfoo::A someAs;
The current Geany correctly parses this and shows the correct thing in the symbols pane, so it has the data, it doesn't even need to use a plugin for the data.
int i = someAs[1]. // Ok TM show me whatcha got, whats legal here?
Answer: a and b because the vector returns a reference to its template parameter, here foo::A that has members a and b, you have to expand the template to find the answer. This is used all over the standard library.
f( // no calltip because no f is defined in this namespace f(anA, // so thats the calltip now?
Answer: void foo::f(A& anA, int inc) even though f() is in namespace foo, because the first argument is type A which is declared in namespace foo, ADL then will find foo::f(). This is also used in the standard library and many other libraries.
Both of these are C++ specific semantics, (types generated by template instantiation and argument dependent lookup). I don't believe TM should be be expanded to include such knowledge.
[...]
Nobody suggested to encode specific language semantics into Geany. I'm suggesting TM should become more generic, i.e. TMTag should transport as much information as required (even if some of it can't be provided by ctags but only some plugins).
But as noted above you need the language specific knowledge to USE the data too. A generic TM just can't answer the question. There is no problem improving TM, but at some point the question is just too language specific for a geenric TM to handle. So the question becomes "who knows when that is?" and only the language specific plugin knows that TM doesn't know what its doing. The plugin has to be asked first, if it then defers to TM, fine.
Most of the issues don't apply to just C++ but so many other languages as well (e.g. Java). All of them can be improved more easily, not just C++, if Geany can offer a powerful framework for that. But the framework Matthew proposed is not powerful to me. It just evades the current problems simply by moving Geany's deficiencies to the plugin space.
Because to make Geany able to answer the two questions I asked above it must have language specific knowledge coded into it (well at least I know of no other languages with ADL and neither does wikipedia https://en.wikipedia.org/wiki/Argument-dependent_name_lookup).
Certainly some issues don't belong to C++ alone, no language has local declarations and scopes handled by TM, and that would help, but again the specifics of scopes, and lookups is language specific, so you need language specific code in Geany.
Cheers Lex
Best regards
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 31.08.2016 um 13:23 schrieb Lex Trotman:
[...]
But to be able to do 2) and 3) accurately needs more knowledge of each language semantics than is currently available in Geany or tagmanager.
That's right. But it doesn't mean the features should be *entirely* moved into plugin space. TM should be improved to be able to hold sufficient information, and plugins should help by providing that data. But *just* the data, don't offload very specific use cases onto them, this will make us less flexible in the long run. If Geany has the data, we can implement new features inside Geany or non-ft-plugins. Otherwise we would have to modify every single ft-plugin for new features and exclude non-ft-plugins.
The problem isn't having the data, its what you do with the data. To repeat two C++ examples I posted on IRC:
namespace foo { struct A { int a=1; int b=2; }; void f(A& anA, int inc){ anA.a+= inc; } };
foo::A anA; std::vectorfoo::A someAs;
The current Geany correctly parses this and shows the correct thing in the symbols pane, so it has the data, it doesn't even need to use a plugin for the data.
int i = someAs[1]. // Ok TM show me whatcha got, whats legal here?
Answer: a and b because the vector returns a reference to its template parameter, here foo::A that has members a and b, you have to expand the template to find the answer. This is used all over the standard library.
f( // no calltip because no f is defined in this namespace f(anA, // so thats the calltip now?
Answer: void foo::f(A& anA, int inc) even though f() is in namespace foo, because the first argument is type A which is declared in namespace foo, ADL then will find foo::f(). This is also used in the standard library and many other libraries.
Both of these are C++ specific semantics, (types generated by template instantiation and argument dependent lookup). I don't believe TM should be be expanded to include such knowledge.
Why not? TM could have separate tags for each known template instance and know which of these applies to someAs, based on information provided by a ft-plugin. Then the rest would work.
Besides, template instantiation requires compiling the units IIUC, which is probably not gonna happen ever? At least not on every keystroke as done currently.
Best regards.
Why not? TM could have separate tags for each known template instance and know which of these applies to someAs, based on information provided by a ft-plugin. Then the rest would work.
Yes, if Geany can be made to understand that `vector<int>` and `vector< int >` and `vector <int >` are all the same type so it will look them up correctly. In parameterised generics, what used to be a type name is now a type syntax, and no matter how I type it, its the same.
As an example Geany trying to lookup something like `template_name<params>::member` to get the type of `member` has to apply syntax analysis to the `template_name<params>` part and can no longer just get a name.
And parameterised generics are available in a number of languages, but with differing syntax (`name{params}` is a popular one for non-{} languages) and again with differing semantics. So thats even more language specific knowledge to be encoded in Geany, either in TM or in Geany if TM doesn't provide the facility.
Much easier to ask the language specific plugin what `template_name<params>::member` or `type_name{params}` is.
Besides, template instantiation requires compiling the units IIUC, which is probably not gonna happen ever? At least not on every keystroke as done currently.
Well, analysing them yes, and it will probably have to happen in a separate thread/process, so again it isn't something thats going to be built-in to Geany.
Best regards.
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
On 2016-08-31 06:30 AM, Thomas Martitz wrote:
Am 31.08.2016 um 13:23 schrieb Lex Trotman:
[...]
Both of these are C++ specific semantics, (types generated by template instantiation and argument dependent lookup). I don't believe TM should be be expanded to include such knowledge.
Why not? TM could have separate tags for each known template instance and know which of these applies to someAs, based on information provided by a ft-plugin. Then the rest would work.
Besides, template instantiation requires compiling the units IIUC, which is probably not gonna happen ever? At least not on every keystroke as done currently.
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
Cheers, Matthew Brush
Am 31.08.2016 um 16:39 schrieb Matthew Brush:
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them from?
Best regards
On 1 September 2016 at 00:43, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 16:39 schrieb Matthew Brush:
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them from?
Libclang needs the full-fat build knowledge that "project" systems on other IDEs provide. Another reason to keep it separate from Geany.
Best regards
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 31.08.2016 um 16:52 schrieb Lex Trotman:
On 1 September 2016 at 00:43, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 16:39 schrieb Matthew Brush:
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them from?
Libclang needs the full-fat build knowledge that "project" systems on other IDEs provide. Another reason to keep it separate from Geany.
So one would have to adjust the build settings and one or more ft-plugins all the time?
IMO, complex build system integration is out of scope for ft-plugins. But I see that not all features can be supported properly without build system.
Best regards
On 1 September 2016 at 00:55, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 16:52 schrieb Lex Trotman:
On 1 September 2016 at 00:43, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 16:39 schrieb Matthew Brush:
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them from?
Libclang needs the full-fat build knowledge that "project" systems on other IDEs provide. Another reason to keep it separate from Geany.
So one would have to adjust the build settings and one or more ft-plugins all the time?
I am guessing that one of the reasons Matthew mentioned build settings access in #1195 is so that the plugins with complex build requirements can set Geany's simple build system to match their complex one.
IMO, complex build system integration is out of scope for ft-plugins. But I see that not all features can be supported properly without build system.
I am not sure I understand you? That sounds like you are saying that plugins cannot be allowed to have complex build systems inside the plugin for its own use, or do I misunderstand?
Best regards _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
On 2016-08-31 07:55 AM, Thomas Martitz wrote:
Am 31.08.2016 um 16:52 schrieb Lex Trotman:
On 1 September 2016 at 00:43, Thomas Martitz kugel@rockbox.org wrote:
Am 31.08.2016 um 16:39 schrieb Matthew Brush:
I can't speak to all compiler libraries, but at least libclang, libpython and libvala "compile" the source (well just the front-end of the compiler is needed). They literally use the built-in compiler front ends to understand the code. In the case of libclang, it additionally provides helpful methods for performing IDE-related features on the AST, while say with libvala, the ft-plugin would be required to perform it's own analysis of the AST to implement those feaures, which is still a lot less work than in Geany/TM since it has access to the full AST/context and the ft-plugin need not fear encoding language specific semantics into its logic.
How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them from?
Libclang needs the full-fat build knowledge that "project" systems on other IDEs provide. Another reason to keep it separate from Geany.
So one would have to adjust the build settings and one or more ft-plugins all the time?
One would have to tell the ft-plugin how to compile the code (where applicable). libclang can read a so-called "compilation-database"[0] which can be generated by CMake[1], Ninja[2], a Make wrapper called Bear[3], or written by hand.
IMO, complex build system integration is out of scope for ft-plugins.
I generally agree, although we could potentially look at improving projects/build commands in the future to some extent, I don't think we should look at it much at this point with ft-plugins.
But I see that not all features can be supported properly without build system.
I don't think they'll usually require a "build system" per se, but they definitively need to be told how to compile the code where applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and libclang (if not using compile_commands.json) they require the `argv` you'd pass to the compiler itself. It may very well be possible for ft-plugins using libraries with the `argv` approach to just grab the compile command from Geany's build commands and convert it back into an `argv` array or something.
Cheers, Matthew Brush
[0]: http://clang.llvm.org/docs/JSONCompilationDatabase.html [1]: https://cmake.org/cmake/help/v3.5/variable/CMAKE_EXPORT_COMPILE_COMMANDS.htm... [2]: https://ninja-build.org/manual.html#_extra_tools [3]: https://github.com/rizsotto/Bear
Am 01.09.2016 um 02:50 schrieb Matthew Brush:
I don't think they'll usually require a "build system" per se, but they definitively need to be told how to compile the code where applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and libclang (if not using compile_commands.json) they require the `argv` you'd pass to the compiler itself. It may very well be possible for ft-plugins using libraries with the `argv` approach to just grab the compile command from Geany's build commands and convert it back into an `argv` array or something.
Not sure how you use Geany, but I never set the cc command line for individual files. I never type CFLAGS or CXXFLAGS into Geany's build settings.
Actually most of the time I don't set up the build system at all and alt-tab to a terminal where I use my employers crazy build system which uses make under the hood but can't be used from within Geany properly.
Then, some other time (e.g. for compiling Geany) I sometimes use the make command. But obviously Geany only knows that running "make" builds something. It doesn't have enough information to construct a compiler command line.
Some other time again I'm working on cross compiled projects. Here I again don't usually use Geany's build system support, but if I would it would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not known to clang (is there even a clang port for avr? I don't think so).
So a ft-plugin that only works if a file can be compiled is likely to be unworkable for me. It needs to deal with when it can't build the source files (either because of lacking *FLAGS or because the target isn't supported by the compiler). Likewise, if a ft-plugin only supports x86 (i.e. is arch dependent) it's not interesting to me.
There is also the possibility that the file I'm editing is not compiled at all with the current build settings, e.g. win32.c.
How would you deal with all that? The ft-plugin has to do something in such situations doesnt it?
Best regards.
On 2016-08-31 09:57 PM, Thomas Martitz wrote:
Am 01.09.2016 um 02:50 schrieb Matthew Brush:
I don't think they'll usually require a "build system" per se, but they definitively need to be told how to compile the code where applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and libclang (if not using compile_commands.json) they require the `argv` you'd pass to the compiler itself. It may very well be possible for ft-plugins using libraries with the `argv` approach to just grab the compile command from Geany's build commands and convert it back into an `argv` array or something.
Not sure how you use Geany, but I never set the cc command line for individual files. I never type CFLAGS or CXXFLAGS into Geany's build settings.
Yeah, that's why the absolute base functionality provided must always exist. I'd probably be more willing to do a little setup for actual projects if there was a benefit though.
Actually most of the time I don't set up the build system at all and alt-tab to a terminal where I use my employers crazy build system which uses make under the hood but can't be used from within Geany properly.
This use-case should continue to be the default supported one.
Then, some other time (e.g. for compiling Geany) I sometimes use the make command. But obviously Geany only knows that running "make" builds something. It doesn't have enough information to construct a compiler command line.
Indeed. To provide "real IDE" features and project requires more than what Geany provides out of the box. I haven't really proposed a design for this, so as it stands ft-plugins would have to provide some means (ex. advanced project support, compile_commands.json, custom GUI, etc) to get this info from the user.
Some other time again I'm working on cross compiled projects. Here I again don't usually use Geany's build system support, but if I would it would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not known to clang (is there even a clang port for avr? I don't think so).
For an example libclang plugin, clang would have to be able to parse the source. It would require valid C/C++/Obj-C code for sure (though IIRC some of the helper functions for IDEs/tools handle partial source files or simple lexing to handle on-the-fly IDE highlighting and such).
So a ft-plugin that only works if a file can be compiled is likely to be unworkable for me. It needs to deal with when it can't build the source files (either because of lacking *FLAGS or because the target isn't supported by the compiler). Likewise, if a ft-plugin only supports x86 (i.e. is arch dependent) it's not interesting to me.
For actual projects you would work on for a while, it would be worth setting up some meta-stuff.
There is also the possibility that the file I'm editing is not compiled at all with the current build settings, e.g. win32.c.
How would you deal with all that? The ft-plugin has to do something in such situations doesnt it?
Fallback to the current features, ideally.
Cheers, Matthew Brush
Am 01.09.2016 um 07:17 schrieb Matthew Brush:
On 2016-08-31 09:57 PM, Thomas Martitz wrote:
Am 01.09.2016 um 02:50 schrieb Matthew Brush:
I don't think they'll usually require a "build system" per se, but they definitively need to be told how to compile the code where applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and libclang (if not using compile_commands.json) they require the `argv` you'd pass to the compiler itself. It may very well be possible for ft-plugins using libraries with the `argv` approach to just grab the compile command from Geany's build commands and convert it back into an `argv` array or something.
Not sure how you use Geany, but I never set the cc command line for individual files. I never type CFLAGS or CXXFLAGS into Geany's build settings.
Yeah, that's why the absolute base functionality provided must always exist. I'd probably be more willing to do a little setup for actual projects if there was a benefit though.
Actually most of the time I don't set up the build system at all and alt-tab to a terminal where I use my employers crazy build system which uses make under the hood but can't be used from within Geany properly.
This use-case should continue to be the default supported one.
But this can't be supported by a libclang ft-plugin, can it? Are you suggesting the default use-case bypasses the ft-plugin? That seems weird to begin with.
Then, some other time (e.g. for compiling Geany) I sometimes use the make command. But obviously Geany only knows that running "make" builds something. It doesn't have enough information to construct a compiler command line.
Indeed. To provide "real IDE" features and project requires more than what Geany provides out of the box. I haven't really proposed a design for this, so as it stands ft-plugins would have to provide some means (ex. advanced project support, compile_commands.json, custom GUI, etc) to get this info from the user.
A solution for this should be part of the discussion, otherwise we're going end up with a dozen completely ways and UIs configure ft-plugins. Which is going to be a PITA for users if the ft-plugins have to be regularly reconfigured on a per-project basis.
Some other time again I'm working on cross compiled projects. Here I again don't usually use Geany's build system support, but if I would it would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not known to clang (is there even a clang port for avr? I don't think so).
For an example libclang plugin, clang would have to be able to parse the source. It would require valid C/C++/Obj-C code for sure (though IIRC some of the helper functions for IDEs/tools handle partial source files or simple lexing to handle on-the-fly IDE highlighting and such).
So a ft-plugin that only works if a file can be compiled is likely to be unworkable for me. It needs to deal with when it can't build the source files (either because of lacking *FLAGS or because the target isn't supported by the compiler). Likewise, if a ft-plugin only supports x86 (i.e. is arch dependent) it's not interesting to me.
For actual projects you would work on for a while, it would be worth setting up some meta-stuff.
I'm afraid I won't be able. The CFLAGS depend on the source file I edit. There are multiple projects involved (e.g. Linux kernel and user space), there is not just the One Project.
The more I'm thinking about it...if key functionality depends on being able to compile the file, I won't be able to use a libclang based ft-plugin.
Really, this is one reason why I use Geany in the first place. Other IDEs such as eclipse require extensive, per-project build setup, to give even basic symbol parsing support. Such IDEs are generally limited to select work flows, such as Makefile-based, which are often incompatible with mine e.g. the work flow at my workplace. I hugely appreciate that Geany gives actually (IMO) decent auto-completion and calltips out of the box (and can be improved with just setting a few file patterns with ProjectOrganizer), for any file I throw at it.
I do see that this is valuable for other people and languages so I'm not fundamentally opposed to do actual compiling (if done right). But there needs to be a decent fallback when this is not possible, even if it just means to fallback to Geany's builtin support (so I'd likely always fall back).
Best regards
On 2016-08-31 11:42 PM, Thomas Martitz wrote:
Am 01.09.2016 um 07:17 schrieb Matthew Brush:
On 2016-08-31 09:57 PM, Thomas Martitz wrote:
Am 01.09.2016 um 02:50 schrieb Matthew Brush:
I don't think they'll usually require a "build system" per se, but they definitively need to be told how to compile the code where applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and libclang (if not using compile_commands.json) they require the `argv` you'd pass to the compiler itself. It may very well be possible for ft-plugins using libraries with the `argv` approach to just grab the compile command from Geany's build commands and convert it back into an `argv` array or something.
Not sure how you use Geany, but I never set the cc command line for individual files. I never type CFLAGS or CXXFLAGS into Geany's build settings.
Yeah, that's why the absolute base functionality provided must always exist. I'd probably be more willing to do a little setup for actual projects if there was a benefit though.
Actually most of the time I don't set up the build system at all and alt-tab to a terminal where I use my employers crazy build system which uses make under the hood but can't be used from within Geany properly.
This use-case should continue to be the default supported one.
But this can't be supported by a libclang ft-plugin, can it? Are you suggesting the default use-case bypasses the ft-plugin? That seems weird to begin with.
It could fall-back to the default functionality if no ft-plugin performs a given feature.
Then, some other time (e.g. for compiling Geany) I sometimes use the make command. But obviously Geany only knows that running "make" builds something. It doesn't have enough information to construct a compiler command line.
Indeed. To provide "real IDE" features and project requires more than what Geany provides out of the box. I haven't really proposed a design for this, so as it stands ft-plugins would have to provide some means (ex. advanced project support, compile_commands.json, custom GUI, etc) to get this info from the user.
A solution for this should be part of the discussion, otherwise we're going end up with a dozen completely ways and UIs configure ft-plugins. Which is going to be a PITA for users if the ft-plugins have to be regularly reconfigured on a per-project basis.
It's going to be plugin/language specific anyways. I agree it should be considered eventually, but it's not required to provide basic hooks for ft-plugins to start providing some features.
Some other time again I'm working on cross compiled projects. Here I again don't usually use Geany's build system support, but if I would it would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not known to clang (is there even a clang port for avr? I don't think so).
For an example libclang plugin, clang would have to be able to parse the source. It would require valid C/C++/Obj-C code for sure (though IIRC some of the helper functions for IDEs/tools handle partial source files or simple lexing to handle on-the-fly IDE highlighting and such).
So a ft-plugin that only works if a file can be compiled is likely to be unworkable for me. It needs to deal with when it can't build the source files (either because of lacking *FLAGS or because the target isn't supported by the compiler). Likewise, if a ft-plugin only supports x86 (i.e. is arch dependent) it's not interesting to me.
If your code can't be (cross) compiled, I'm afraid you're screwed anyway :)
For some stuff, like partial code (ex. you're still typing it) or guarded out by platform macros, I'd expect some some reduced functionality, or different behaviour, at least.
To test it out, just try a competent full-fledge IDE such as QtCreator, Visual Studio, XCode, or Eclipse (among many others) and see what it does.
For actual projects you would work on for a while, it would be worth setting up some meta-stuff.
I'm afraid I won't be able. The CFLAGS depend on the source file I edit. There are multiple projects involved (e.g. Linux kernel and user space), there is not just the One Project.
If you can't describe your build, yeah it's not so useful.
The more I'm thinking about it...if key functionality depends on being able to compile the file, I won't be able to use a libclang based ft-plugin.
In order to provide "smart" features requires to understand the code. If you can't describe your build and make a regular c/c++/obj-c (cross) compiler able to handle your code, I'm afraid at least my CDK/libclang ft-plugin indeed wouldn't be of use to you. Doesn't mean others used to normal IDEs with describable builds won't benefit though.
Really, this is one reason why I use Geany in the first place. Other IDEs such as eclipse require extensive, per-project build setup, to give even basic symbol parsing support. Such IDEs are generally limited to select work flows, such as Makefile-based, which are often incompatible with mine e.g. the work flow at my workplace. I hugely appreciate that Geany gives actually (IMO) decent auto-completion and calltips out of the box (and can be improved with just setting a few file patterns with ProjectOrganizer), for any file I throw at it.
There is no desire to remove this base functionality, only to enhance it for people who want more than you from Geany without bloating the core.
I do see that this is valuable for other people and languages so I'm not fundamentally opposed to do actual compiling (if done right). But there needs to be a decent fallback when this is not possible, even if it just means to fallback to Geany's builtin support (so I'd likely always fall back).
That would be ideal.
Cheers, Matthew Brush
Le 29/08/2016 à 05:14, Matthew Brush a écrit :
[…]
Syntax Highlighting
Most likely using an API based on/similar to Scintilla's "container lexers".
At the minimum, it could have a callback something like:
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*, guint start_pos, guint end_pos, gpointer user_data);As with Scintilla's "container lexer", it would just tell the provider what and where to highlight. It might be pointless providing `end_pos` it could probably just highlight a whole line at time (maybe like Scintilla's 'style-needed' notification).
I'm really not sure it's a good idea to go the custom callback way. IMO, we should first try and see how easy it'd be with plugins providing their own full-blown Scintilla lexer library that we just add and use.
Having our own callback means one more indirection, and changing the SciLexer to CONTAINER anyway, so I don't see much advantage just now.
On 2016-08-30 06:43 AM, Colomban Wendling wrote:
Le 29/08/2016 à 05:14, Matthew Brush a écrit :
[…]
Syntax Highlighting
Most likely using an API based on/similar to Scintilla's "container lexers".
At the minimum, it could have a callback something like:
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*, guint start_pos, guint end_pos, gpointer user_data);As with Scintilla's "container lexer", it would just tell the provider what and where to highlight. It might be pointless providing `end_pos` it could probably just highlight a whole line at time (maybe like Scintilla's 'style-needed' notification).
I'm really not sure it's a good idea to go the custom callback way. IMO, we should first try and see how easy it'd be with plugins providing their own full-blown Scintilla lexer library that we just add and use.
The only positive I really see, which in practice probably won't exist, is modularity and ability to re-use lexers independent of Geany/ft-plugin (ie. for all Scintilla-using apps). I say in practice because at least with my `LexClang.so` I needed it to be bound into Geany anyway to get hooks for when to re-parse (you can't re-parse a million token C++ file each time Scintilla wants to re-colour a line of code). Further, the dynamic lexer needs to cooperate with Geany/ft-plugin, or at least deviate from normal Scintilla lexers, if it wanted to provide/setup its own lexical states/styles (TBD how this part will go).
Having our own callback means one more indirection, and changing the SciLexer to CONTAINER anyway, so I don't see much advantage just now.
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Cheers, Matthew Brush
All I really wanted was a way to disable
Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
In fact some of the tools using scintilla and which provide a richer styling do exactly that, probably for all the reasons Matthew notes.
Cheers, Matthew Brush
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
On 31 August 2016 at 11:27, Matthew Brush mbrush@codebrainz.ca wrote:
On 2016-08-30 06:43 AM, Colomban Wendling wrote:
Le 29/08/2016 à 05:14, Matthew Brush a écrit :
[…]
Syntax Highlighting
Most likely using an API based on/similar to Scintilla's "container lexers".
At the minimum, it could have a callback something like:
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*, guint start_pos, guint end_pos, gpointer user_data);As with Scintilla's "container lexer", it would just tell the provider what and where to highlight. It might be pointless providing `end_pos` it could probably just highlight a whole line at time (maybe like Scintilla's 'style-needed' notification).
I'm really not sure it's a good idea to go the custom callback way. IMO, we should first try and see how easy it'd be with plugins providing their own full-blown Scintilla lexer library that we just add and use.
The only positive I really see, which in practice probably won't exist, is modularity and ability to re-use lexers independent of Geany/ft-plugin (ie. for all Scintilla-using apps). I say in practice because at least with my `LexClang.so` I needed it to be bound into Geany anyway to get hooks for when to re-parse (you can't re-parse a million token C++ file each time Scintilla wants to re-colour a line of code). Further, the dynamic lexer needs to cooperate with Geany/ft-plugin, or at least deviate from normal Scintilla lexers, if it wanted to provide/setup its own lexical states/styles (TBD how this part will go).
Having our own callback means one more indirection, and changing the SciLexer to CONTAINER anyway, so I don't see much advantage just now.
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Agree with Matthew.
The point of not using the standard Scintilla lexer is to be able to add semantic information made available for the language support library in the FT-Plugin.
The major example of this is fixing the current situation where any name that is a type in any scope is coloured as a type in every other scope, even if the type is not visible there.
That means that the lexer is intimately tied into the FT-Plugin so it likely won't run without the FT-Plugin anyway, so actually including it in the plugin and accessing it via the container lexers interface looks better than having a separate DLL that won't run by itself anyway and then has to be sure its loaded at the right time.
Cheers Lex
Cheers, Matthew Brush
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 01.09.2016 um 05:42 schrieb Lex Trotman:
On 31 August 2016 at 11:27, Matthew Brush mbrush@codebrainz.ca wrote:
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Agree with Matthew.
The point of not using the standard Scintilla lexer is to be able to add semantic information made available for the language support library in the FT-Plugin.
The major example of this is fixing the current situation where any name that is a type in any scope is coloured as a type in every other scope, even if the type is not visible there.
That means that the lexer is intimately tied into the FT-Plugin so it likely won't run without the FT-Plugin anyway, so actually including it in the plugin and accessing it via the container lexers interface looks better than having a separate DLL that won't run by itself anyway and then has to be sure its loaded at the right time.
Can this support color schemes and custom keywords (although the latter is probably not very important).
I don't know what SCLEX_CONTAINER is.
Best regards
On 2016-08-31 10:00 PM, Thomas Martitz wrote:
Am 01.09.2016 um 05:42 schrieb Lex Trotman:
On 31 August 2016 at 11:27, Matthew Brush mbrush@codebrainz.ca wrote:
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Agree with Matthew.
The point of not using the standard Scintilla lexer is to be able to add semantic information made available for the language support library in the FT-Plugin.
The major example of this is fixing the current situation where any name that is a type in any scope is coloured as a type in every other scope, even if the type is not visible there.
That means that the lexer is intimately tied into the FT-Plugin so it likely won't run without the FT-Plugin anyway, so actually including it in the plugin and accessing it via the container lexers interface looks better than having a separate DLL that won't run by itself anyway and then has to be sure its loaded at the right time.
Can this support color schemes and custom keywords (although the latter is probably not very important).
I don't know what SCLEX_CONTAINER is.
It's the built-in (application-provided) lexing support in Scintilla. It means "Tell me when, and I'll highlight the source for you". It only requires a GTK+ signal from Scintilla rather than having to implement a concrete C++ class of ILexer and conforming to some interface in a separate DLL, to perform roughly the same task.
http://www.scintilla.org/ScintillaDoc.html#SCN_STYLENEEDED
Cheers, Matthew Brush
On 2016-08-31 08:42 PM, Lex Trotman wrote:
On 31 August 2016 at 11:27, Matthew Brush mbrush@codebrainz.ca wrote:
On 2016-08-30 06:43 AM, Colomban Wendling wrote:
[...]
Having our own callback means one more indirection, and changing the SciLexer to CONTAINER anyway, so I don't see much advantage just now.
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Agree with Matthew.
The point of not using the standard Scintilla lexer [...]
I wouldn't even call it "standard" lexer. Though it conforms to the same C++ interface as a few other modern, well-maintained lexers, AFAIK only handful of Scintilla's lexers (ex. LexCpp) are actually able to compile as dynamic libraries.
One of the few useful dynamic lexers I've ever seen is Scintillua[0] and it actually makes some kind of sense to be a generic dynamic lexer here since it can proxy for other lexers using a completely different non-Scintilla mechanisms internally (ie. Lua and PEGs).
Cheers, Matthew Brush
Le 31/08/2016 à 03:27, Matthew Brush a écrit :
On 2016-08-30 06:43 AM, Colomban Wendling wrote:
Le 29/08/2016 à 05:14, Matthew Brush a écrit :
[…]
I'm really not sure it's a good idea to go the custom callback way. IMO, we should first try and see how easy it'd be with plugins providing their own full-blown Scintilla lexer library that we just add and use.
The only positive I really see, which in practice probably won't exist, is modularity and ability to re-use lexers independent of Geany/ft-plugin (ie. for all Scintilla-using apps). I say in practice because at least with my `LexClang.so` I needed it to be bound into Geany anyway to get hooks for when to re-parse (you can't re-parse a million token C++ file each time Scintilla wants to re-colour a line of code). Further, the dynamic lexer needs to cooperate with Geany/ft-plugin, or at least deviate from normal Scintilla lexers, if it wanted to provide/setup its own lexical states/styles (TBD how this part will go).
Having our own callback means one more indirection, and changing the SciLexer to CONTAINER anyway, so I don't see much advantage just now.
With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to fit well (too isolated, too many assumptions that it's a simple dumb lexer and not a semantic-based on, etc) . All I really wanted was a way to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the filetype in Geany, and without doing it behind Geany's back from the plugin.
Okay then, if there's good reasons to do so, fine.
I just thought that it would be handier if later on plugins want to be able to add new filetypes altogether (e.g. not necessarily based on lib<language>, but just filetypes), but maybe not, or maybe it's not good for all cases.
Regards, Colomban