#592 #1457 #1294 #1188 #1187 #1195 and others all raise the issue of plugins providing language services to Geany.
There have been various discussions in the past on the subject with no conclusion. So its time to approach it again and this issue is intended to do that (without hijacking any of the above issues/PRs).
Note the purpose is first to DESIGN the API so that it can be implemented progressively without spamming committers with huge changes to core Geany, and without the first plugin to use any of the API having to implement far more than it needs to.
Since I come from an Aerospace background let me initiate the formal requirements-design-implement process that I am used to, by suggesting some terminology and then requirements:
# Terminology
- A "feature" is something Geany does that is or could be language specific, eg autocomplete, indentation, styling. When referring to a "feature" below it is understood that it is specific to a filetype unless otherwise noted.
- A "filetype support plugin" is a plugin that provides some support for one or more features for the programming language the filetype supports. (Whilst most of the rest of the world calls it "language" support, Geany calls the language specific things "filetypes" so lets keep it consistent).
- There are two steps in starting a plugin "loading" where information needed for the plugin manager dialog is gathered and "activation" when the user selects the plugin in the dialog.
- A "request" is an interaction initiated by Geany for support from the plugin (eg by calling a plugin function)
- A "reply" is the data sent back as a result of a "request" (eg as function return data)
- A "notification" is an interaction initiated by the plugin (eg by calling a Geany function)
# Requirements
- On activation (not loading), filetype support plugins must be able to inform Geany of the service(s) they provide and the filetypes(s) those services are for.
- Geany should provide at least the current capabilities for features that no plugin has offered to provide.
- On de-activation, filetype support plugins must be able to inform Geany that their services are no longer available, however it is probably better if Geany can automatically handle loss of the services and fall back to its default behaviour.
- Currently Geany does not mediate any resources shared between plugins, so it is not proposed that Geany will mediate between multiple filetype support plugins trying to provide the same feature, it can be UB if first wins, last wins or something else, except Geany should not crash.
- The default Geany behaviour for some features may not need to be totally replaced by the plugin, just tweaked. The API must be designed to allow both behaviours, with the plugin specifying which it provides for each reply.
- For all features, Geany should always allow replacement of its default behaviour by a plugin, Geany does not know what weirdnesses programming languages may come up with and should not enforce rules from one language on another where it may not be appropriate. (C++ is NOT C, and Haskell even less :)
- It is up to the implementation of the feature in Geany if it allows tweaking, and what and how that is communicated from the plugin
# Preliminary Design Notes
This is NOT a complete design, just a start.
- My first thought was to use GTK signals, its probably not appropriate to have a signal for each feature for each language, but it is possible for there to be one signal per feature and the plugin callbacks then need to check the filetype of the current document and return false if its not one they handle, so the next plugin callback can be invoked. Given that an individual user probably won't invoke a large number of filetype plugins at once this may be acceptable, even with the additional overhead if the plugins are in languages other than C (eg Python).
- I am not enough of a G* expert to comment on how simple passing data from and to signal callbacks is, for example whilst features like indentation or formatting can just manipulate the Scintilla buffer, for features like autocomplete the plugin needs to return a list of options.
- Since Geany is single threaded and not re-entrant, notifications realistically can only be provided from asynchronous operations occurring in other threads via the mainloop idle-add mechanism.
Feel free to expand this further.
Current codebase has a lot of `if` branches checking for specific languages, approaching an implementation of this design would likely mean that those hard-coded checks would want to move to plugin-based approach; that is to be recognized as **some** work.
Additionally to what can be found by looking at the aforementioned checks, we could look at how language-specific plugins have been designed&implemented in other notable editors.
Finally, the design should not be too broad to conflict with the "Lite" nature of Geany, and that is surely a challenge as well.
I recognize the problem with callbacks and asynchronous replies, and I can foresee a situation where Geany gets "stuck" because of plugins (so extra debugging/development features are probably needed to accommodate for the variability introduced by such plugins). You can probably already see some of these problems with "custom actions". I have no simple solution for this either.
What do we do with the mentioned issues/PRs? #1187 and #1195 look particularly relevant
Current codebase has a lot of if branches checking for specific languages, approaching an implementation of this design would likely mean that those hard-coded checks would want to move to plugin-based approach; that is to be recognized as some work.
The key thing is the comment above that the design must allow progressive implementation, the current capability is the fallback, and is not intended to be removed for a long time. The intention is to design a mechanism so that no more language specific branches need get added to Geany core. But unless the process of including plugin capability in place of the built-ins can proceed in small chunks it probably isn't going to happen at all, simply due to resource availability.
we could look at how language-specific plugins have been designed&implemented in other notable editors.
Certainly, if anyone wants to chime in with their knowledge (or investigations) of existing editors that will be useful and may be adaptable to Geany.
and I can foresee a situation where Geany gets "stuck" because of plugins
Plugins are part of the Geany process and address space, so if they crash or hang so does Geany, thats an unfortunate fact of life. It is very unlikely that will change so plugin devs just have to be careful.
design must allow progressive implementation [...] The intention is to design a mechanism so that no more language specific branches need get added to Geany core. But unless the process of including plugin capability in place of the built-ins can proceed in small chunks it probably isn't going to happen at all, simply due to resource availability.
Couldn't agree more, so I'd say let's staple this approach as the way to go.
Certainly, if anyone wants to chime in with their knowledge (or investigations) of existing editors that will be useful and may be adaptable to Geany.
I will check around and report back if I find anything comparable/useful.
Plugins are part of the Geany process and address space, so if they crash or hang so does Geany, thats an unfortunate fact of life. It is very unlikely that will change so plugin devs just have to be careful.
Yeah, this is a too fundamental aspect of Geany and I would not even think of changing it in any way (IMO it is the basis of being "Lite" and fast too). However, the moment a corpus of new plugins emerges (external or official) we might want to define testsuites to catch rogue plugin behavior situations and not necessarily at UI-level as I was hinting in #1462.
The reason I am mentioning this is - from my tiny experience with #1457 - that the moment I coded a synchronous call to an external command I knew a "sin" was being done, in the sense that potentially Geany would become stuck when the external process becomes rogue. And as you said, no safeguard would be in place for this even with the plugins system.
At least most plugins don't affect Geany stability unless they are activated, so the impact of a "rogue" plugin is confined to those who use it, and they can turn it off. (Note, the plugin DLL is loaded and some code run to fill in the Plugin Manager dialog, but plugins shouldn't do anything major unless activated)
Definitely plugins should not delay the main thread, in particular blocking IO to anything slow, even if it doesn't stop. Plugins should use the Glib asynchronous IO facilities or separate threads (but note Geany is not thread safe, so you can't call Geany functions from those threads).
This last makes the issue of subprocesses being used to answer autocomplete or formatting or whatever somewhat problematical. Geany should not be blocked until the plugin gets an answer, but what should it do in the meantime, and its not re-entrant, so injecting replies later becomes problematical.
This last was one of the major blocks to previous efforts, no good asynchronous solution has been found yet.
And the alternative of using the subprocess to supply the symbol information for the Geany symbol management, from which Geany can quickly create the autocomplete list, hits the stumbling block that the symbol management has only a subset of symbol table functionality (most specifically Geany has no lexical scope handling or include file tracking), so accuracy is lost, and many irrelevant symbols get included in the autocomplete list.
And similar concerns apply to things like accurate indentation determination, or formatting.
The problem you are describing is familiar to me from experience with D/Go and interfacing X11 and OpenGL subsystems (or even sound systems, sometimes): regular C libraries/software expect the application to keep calling from the same thread it originally initiated the interaction from.
The way this is addressed in Go (which has no native threads concept) for example is by doing the (non-blocking) multiplexing of async signals in the main thread, while having all the "action" happen elsewhere. For reference, see https://github.com/golang/go/wiki/LockOSThread.
How such centralized event-based approach would play with existing Geany functionality, that I can't tell and I do believe its limitations have probably already become evident from the attempts you mentioned.
How such centralized event-based approach would play with existing Geany functionality, that I can't tell and I do believe its limitations have probably already become evident from the attempts you mentioned.
It works fine, GLib has async IO facilities designed to work with its main loop. It also has a way to safely send messages from worker threads to the GUI thread if the normal async IO approach wasn't used.
@codebrainz good to hear. Then I guess some more detail should be put into this spec and then implementation attempts can be started?
@gdm85 sure, ideally using the existing mailing list thread for the original discussion linked from the original Issue tracking this #1195. I think we have reached a basic design in that thread (or at least one has been proposed), but I have doubts we'll ever get it implemented due to lack of consensus and discussion bogging down what is actually not very difficult to implement.
If you're interested, I have a branch somewhere where I've implemented stuff from the discussions, I can try and track it down for you.
@codebrainz let me guess, `plugin-api` at 7296d467f38b9bffa2305282b5da6b9c2a165058? Didn't check the commits there yet.
I have to read the material in #1195 and mailing list to get up to date; I share your view on complexity, I will post back later after reading around.
@gdm85 I'm not sure if I pushed it onto my Github fork, it might be local only.
@gdm85 I just pushed what I had locally, see my `ft-plugin/*` branches. I believe the only interesting one is `ft-plugins/experiments-3` and `ft-plugins/gui` which is based on it.
My first thought was to use GTK signals, its probably not appropriate to have a signal for each feature for each language, but it is possible for there to be one signal per feature and the plugin callbacks then need to check the filetype of the current document and return false if its not one they handle, so the next plugin callback can be invoked
Signals are a good start, but maybe too heavy-weight for the purpose.
I'd rather like to see interfaces (GObject-based) to be used. They are as powerful (if not more, since you can attach data with instances), but also allow for potential adoption of libpeas (in a future far away) which uses interfaces as natural extension points.
In some way, libpeas solves already exactly what we want here: plugins can extend or replace core functionality by implementing interfaces, which are registered in the core application and then instantiated when necessary. Because of this, our design should be compatible with that.
IIRC in the original discussion people were against using the normal GObject approach, so the design proposal used hook functions more like already in use inside Geany. +1 for using interfaces, but I doubt you'll get much buy-in from others.
That's +2, from those that potentially would also do (some of) the work.
The problem with plain hooks is that you always only support C/native plugins. With interfaces (signals too, to be fair) python plugins could easily work directly without extra native code portions.
@kugel- can you explain what you mean by signals being too "heavy weight"? Its the way existing events in Geany are notified to plugins and anyone else who is interested ("document-save" etc). But signal data passing and return is poor.
@kugel- your point about making the facilities available to non-C plugins is good.
The problem I see with anything gobject based is it requires specialist knowledge for someone to contribute a change to Geany to make a function overridable/modifyable by a plugin. I would suspect that few plugin devs will have that ability, so making changes to allow plugins to do stuff will bottleneck at the few internal project people who have that knowledge.
It would be nice if there was a way that worked as simply as using signals, ie emit() and connect(). Then more people could contribute changes to Geany that let their plugin modify built-in behaviour.
What do you mean by "specialist knowledge"? GObject interfaces are a fundamental part of GObject (and GLib in general), just like signals.
Perhaps you're more fluent with signals, but GObject interfaces is *far* from "specialist knowledge" if you're working on a glib application anyway.
To make a core feature plugin-replacable you have to change Geany anyway. Using either singals or interfaces requires knowledge about the GObject fundamentals.
signals are heavy weight because: - multiple options exist: connecting before, after, with object data (or without), with closure (or without). selecting the right one isn't always trivial. - from follows that dispatching is complex (especially with potentially multiple handlers), whereas interfaces boil down to a single vtable lookup much like C++ virtual functions. - glib has lots of extra code to handle blocked or recursive signals - connecting via string, which requires runtime look ups in various hash tables - you always have extra code that connects, in addition to the callback itself - passing arbitrary data to the callback is nasty, most of the time you need to malloc per-callback data structures
In addition they are tricky to use because: - most errors can only be observed at runtime (or even later), like connecting in an inappropriate function or connecting to the wrong signal (no compile time diagnostics support) - sutiable function signatures are not readily available, since the type of callback function isn't declared anyway
Signals are really meant for event-based programming. It can be used for extensibility purposes too, yes, but they are really overpowered for the task. Interfaces are much more lightweight and easier to use correctly (IMO).
BTW: We have the classbuilder plugin for setting up GObjects, it could be extended for interfaces.
What do you mean by "specialist knowledge"? GObject interfaces are a fundamental part of GObject (and GLib in general), just like signals.
Sorry not to be clear, DEFINING new GObjects is what is not so common, its no longer even in Glib.
To make a core feature plugin-replacable you have to change Geany anyway. Using either singals or interfaces requires knowledge about the GObject fundamentals.
How do you need any GObject knowledge to emit signals? eg https://github.com/geany/geany/blob/master/src/document.c#L711
Note, I am not advocating for signals specifically, just something that is as easy to use on both sides as signals are. I agree with your observation on passing data but don't think the other inefficiencies matter too much.
Interfaces are much more lightweight and easier to use correctly (IMO).
I will admit lack of familiarity, maybe it would be better if you could show an example, say what would it need in Geany and in the plugin for the plugin to replace Geany's autoindent [functionality](https://github.com/geany/geany/blob/master/src/editor.c#L1504) or just to get the indent [size](https://github.com/geany/geany/blob/master/src/editor.c#L1468)
Sorry not to be clear, DEFINING new GObjects is what is not so common, its no longer even in Glib.
I don't understand that. GObject is part of GLib. GLib makes heavy use of GObjects and interfaces.
How do you need any GObject knowledge to emit signals? eg https://github.com/geany/geany/blob/master/src/document.c#L711
That's just emitting. That doesn't show defining the signal (see geanyobject.c), or the modifications required to a given feature to be replacable or extensible through a signal (or other means).
Calling an interface is equally a one liner (with added compile time checks): `my_interface_method(object);`
I admit that defining an interfaces requires some boilerplate. Signal definitions also require boilerplate, but less of it. Both can be automated with classbuilder or even snippets (both shipped with Geany itself). Part of the interface boilerplate is clear function signature declaration that allow for compile time checks, though.
Also, be sure to look what required if you want to get a return value from a signal emission. For example look at the editor-notify signal. This scenario is definitely a requirement. You need a accumulator and handle multiple signal handler connections.
DEFINING new GObjects is what is not so common
It's extremely common, just not in Geany. It makes it much easier for external contributors when the code is idiomatic for the toolkit (ie. using GObject normally). I've been able to study the code for and contribute to a number of projects easily because they used the toolkit in the usual way instead of doing all kinds of ad hoc things.
I don't think signals will be right for most of this stuff, though I'm sure we'll add some (hopefully not jamming more into the GeanyObject singleton god object). At worst we should use our own NiH virtual functions like we do for the plugin API now, IMO, though as @kugel- mentioned it does make it harder for binding to other languages, even for hand-written bindings like GeanyPy.
That's just emitting. That doesn't show defining the signal (see geanyobject.c),
Ok, plus a call to g_signal_new(), and don't forget the enum member in geanyobject.h ;-D
or the modifications required to a given feature to be replacable or extensible through a signal (or other means).
Those changes will be the required irrespective of the mechanism for the extension.
I admit that defining an interfaces requires some boilerplate.
This is the part I have concerns about. Please show the indentation example I suggested.
I don't think signals will be right for most of this stuff, though I'm sure we'll add some
Like I said and @kugel- agreed, signals are bad at data passing, what we need is to be as simple to define and use as signals, but with better data transfer capabilities.
(hopefully not jamming more into the GeanyObject singleton god object).
GObjectification of Geany needs to be kept separate from this feature or again it will be bottlenecked on the few contributors with appropriate knowledge. Also I would doubt how incremental the changes to do that would be.
At worst we should use our own NiH virtual functions like we do for the plugin API now, IMO, though as @kugel- mentioned it does make it harder for binding to other languages, even for hand-written bindings like GeanyPy.
Yeah, Plugins already have to be able to call Geany functions and define callbacks, so I suggest we can safely assume that those capabilities are available in whatever plugin language. And agree that it would be a pain to manually add lots of functions to any plugin bindings, its all thats available until @kugel- releases his GIR based plugin language support.
Any thoughts on how to handle asynchronism between Geany and the plugin, for example if the indentation example I suggested above was slow in the plugin (eg querying a separate process) how would the interface operate so Geany is not blocked while the plugin is waiting?
github-comments@lists.geany.org