Hello,
I was asked to share a bit about my roadmap regarding plugins. I'll try to give you a better idea with this post.
My ultimate goal is to implement a clean and maintainable way to support non-C plugins, preferably using existing widely used techniques. I concluded that libpeas[1] in conjunction with gobject-introspection can provide the base for this. Since Geany is not at all prepared for this I have several infrastructure which I do want to get merged into Geany. When Geany core is sufficiently setup for this, the non-C plugin enablement can happen outside the core, as a plugin, to stabilize there.
So, here's the set of infrastructure changes for the core. Please let me stress that all of this will happen in backward-compatible manner, no existing plugins break as part of this.
- linkage-cleanup (PR#429) - This changes the way plugins access Geany API functions. Instead of exporting a pointer to a struct of structs of API function pointers, now the APIs are exported directly. This work also includes an effort to stop exporting all function (we do this currently as a workaround to allow gtkbuilder to work), so *only* API function are exported and plugins cannot call internal function anymore. This change is also required to allow gobject-introspection to find geany functions at runtime (through g_module_symbol()) in the future. - new API functions for registering keybindings (PR#376). The current API functions do not allow context information to be stored and passed to the key handler function. This makes it very hard for non-C plugins to use these function. So what's needed are key handlers that get the necessary context information. This allows interprepted language plugins to register keybindings. - A new plugin loader mechanism (a thread about this is already running on the devel list): Similarly to the keybindings, the plugin_* functions implemented by plugins do not carry any context information, making it hard for non-C plugins to implement them properly. Therefore a new loader mechaism is needed so that the context information can be passed. The loader works such that an API function is called to register a function pointer table. This is crucial to possibly support plugins that register other plugins (so called pluxies) which is simply not possible with the current mechaism. The current loader is kept for backwards compatibility (but will not receive new features). - New API functions to allow plugins to act as proxy plugins (pluxies). These pluxies can then implement whatever is needed to execute code in the in the actual plugin, like invoking an interpreter or firing up a java vm. The pluxies call the new loader's API function on behalf of the actual plugin. The API function to implement the pluxies is a simple geany_register_pluxy() that, much like the normal plugin loader, that pluxies use to pass a function pointer table which implements the necessary hooks (probe(), load() and unload())
Once this is in place in the core, my roadmap contains the following items, which are implemented (at least initially) in a plugin, so no further changes to the cure should be necessary. - Modify geanypy to use the new pluxy APIs. This will finally enable geanypy to show the python plugins in the normal PM dialog and support keybindings - Create a new pluxy that supports libpeas-based plugins (codename: peasy). Peasy will use libpeas to load plugins and their metadata. - Part of the peasy work is also work on creating vala and gobject-introspection bindings for Geany's API functions, so that we can support python, javascript and lua out of the box.
This is my roadmap so far. It changed quite a bit since I started this non-C-plugins effort a year ago, but I hope it will be good for everyone. Please share your opinions on this or ask questions.
Best regards.
Thomas,
Thanks for that, it now makes it clearer where you are going, and allows the individual steps to make more sense.
I agree with your four problems (shortened):
- linkage - keybindings - plugin context - allow for proxies
My concern is that the solutions seem complex, though, having followed your attempts over time to address these issues, I do understand that some of the "simple" options don't actually cut it.
Will need to think on it further.
Cheers Lex
On 29 March 2015 at 10:23, Thomas Martitz kugel@rockbox.org wrote:
Hello,
I was asked to share a bit about my roadmap regarding plugins. I'll try to give you a better idea with this post.
My ultimate goal is to implement a clean and maintainable way to support non-C plugins, preferably using existing widely used techniques. I concluded that libpeas[1] in conjunction with gobject-introspection can provide the base for this. Since Geany is not at all prepared for this I have several infrastructure which I do want to get merged into Geany. When Geany core is sufficiently setup for this, the non-C plugin enablement can happen outside the core, as a plugin, to stabilize there.
So, here's the set of infrastructure changes for the core. Please let me stress that all of this will happen in backward-compatible manner, no existing plugins break as part of this.
- linkage-cleanup (PR#429) - This changes the way plugins access Geany API
functions. Instead of exporting a pointer to a struct of structs of API function pointers, now the APIs are exported directly. This work also includes an effort to stop exporting all function (we do this currently as a workaround to allow gtkbuilder to work), so *only* API function are exported and plugins cannot call internal function anymore. This change is also required to allow gobject-introspection to find geany functions at runtime (through g_module_symbol()) in the future.
- new API functions for registering keybindings (PR#376). The current API
functions do not allow context information to be stored and passed to the key handler function. This makes it very hard for non-C plugins to use these function. So what's needed are key handlers that get the necessary context information. This allows interprepted language plugins to register keybindings.
- A new plugin loader mechanism (a thread about this is already running on
the devel list): Similarly to the keybindings, the plugin_* functions implemented by plugins do not carry any context information, making it hard for non-C plugins to implement them properly. Therefore a new loader mechaism is needed so that the context information can be passed. The loader works such that an API function is called to register a function pointer table. This is crucial to possibly support plugins that register other plugins (so called pluxies) which is simply not possible with the current mechaism. The current loader is kept for backwards compatibility (but will not receive new features).
- New API functions to allow plugins to act as proxy plugins (pluxies).
These pluxies can then implement whatever is needed to execute code in the in the actual plugin, like invoking an interpreter or firing up a java vm. The pluxies call the new loader's API function on behalf of the actual plugin. The API function to implement the pluxies is a simple geany_register_pluxy() that, much like the normal plugin loader, that pluxies use to pass a function pointer table which implements the necessary hooks (probe(), load() and unload())
Once this is in place in the core, my roadmap contains the following items, which are implemented (at least initially) in a plugin, so no further changes to the cure should be necessary.
- Modify geanypy to use the new pluxy APIs. This will finally enable geanypy
to show the python plugins in the normal PM dialog and support keybindings
- Create a new pluxy that supports libpeas-based plugins (codename: peasy).
Peasy will use libpeas to load plugins and their metadata.
- Part of the peasy work is also work on creating vala and
gobject-introspection bindings for Geany's API functions, so that we can support python, javascript and lua out of the box.
This is my roadmap so far. It changed quite a bit since I started this non-C-plugins effort a year ago, but I hope it will be good for everyone. Please share your opinions on this or ask questions.
Best regards. _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 29.03.2015 um 05:20 schrieb Lex Trotman:
Thomas,
Thanks for that, it now makes it clearer where you are going, and allows the individual steps to make more sense.
I agree with your four problems (shortened):
- linkage
- keybindings
- plugin context
- allow for proxies
My concern is that the solutions seem complex, though, having followed your attempts over time to address these issues, I do understand that some of the "simple" options don't actually cut it.
Will need to think on it further.
Which of these steps seem complex to you in particular? Perhaps I can explain more detail on those to make things clearer for you. For me, none of these is really that complex, both in terms of conceptual complexity and lines of code changed. The most complex is probably the pluxy part, but still not really complex IMO (it's just a bit of refactoring of the internal load/unload functions functions and an API that allows plugins to add such functions for some filetypes). If anything, complexity is added by maintaining backward compatibility.
My current roadmap is such that it requires the least amount of changes to Geany core while still enabling libpeas-based and other non-C plugins to co-exist with standard plugins and not be second class citizens.
Other simple options (that provide a solution for the above, i.e. not current geanypy) don't exist. In fact, the other solution I have tried brought significant changes to Geany core to support libpeas-plugins directly in the core. Simple isn't always best too, we should strive to make all of this effort future proof but at the same time maintain backward compatibility.
Best regards.
Le 29/03/2015 00:23, Thomas Martitz a écrit :
[…]
- linkage-cleanup (PR#429) - This changes the way plugins access Geany
API functions. Instead of exporting a pointer to a struct of structs of API function pointers, now the APIs are exported directly. This work also includes an effort to stop exporting all function (we do this currently as a workaround to allow gtkbuilder to work), so *only* API function are exported and plugins cannot call internal function anymore. This change is also required to allow gobject-introspection to find geany functions at runtime (through g_module_symbol()) in the future.
Agreed, but on the fact that it's not actually "also required for GI", it's *only* required for this kind of thing. It's good and everything, but not required but for dynamically looking up the functions.
- new API functions for registering keybindings (PR#376). The current
API functions do not allow context information to be stored and passed to the key handler function. This makes it very hard for non-C plugins to use these function. So what's needed are key handlers that get the necessary context information. This allows interprepted language plugins to register keybindings.
Agreed.
- A new plugin loader mechanism (a thread about this is already running
on the devel list): Similarly to the keybindings, the plugin_* functions implemented by plugins do not carry any context information, making it hard for non-C plugins to implement them properly. Therefore a new loader mechaism is needed so that the context information can be passed. The loader works such that an API function is called to register a function pointer table. This is crucial to possibly support plugins that register other plugins (so called pluxies) which is simply not possible with the current mechaism. The current loader is kept for backwards compatibility (but will not receive new features).
Agreed.
- New API functions to allow plugins to act as proxy plugins (pluxies).
These pluxies can then implement whatever is needed to execute code in the in the actual plugin, like invoking an interpreter or firing up a java vm. The pluxies call the new loader's API function on behalf of the actual plugin. The API function to implement the pluxies is a simple geany_register_pluxy() that, much like the normal plugin loader, that pluxies use to pass a function pointer table which implements the necessary hooks (probe(), load() and unload())
That's the part I'm really fuzzy about. I really don't see why we need this specific layer (maybe in an ideal world not bothering about how Geany currently does it): as asked on the other thread, why do we need anything beside geany_plugin_register() (required for everyone) and geany_plugin_unregister() (required only when registered sub-plugins, as the parent need to clean them up when it itself quits)?
Regards, Colomban
Le 29/03/2015 00:23, Thomas Martitz a écrit :
[…]
- linkage-cleanup (PR#429) - This changes the way plugins access Geany
API functions. Instead of exporting a pointer to a struct of structs of API function pointers, now the APIs are exported directly. This work also includes an effort to stop exporting all function (we do this currently as a workaround to allow gtkbuilder to work), so *only* API function are exported and plugins cannot call internal function anymore. This change is also required to allow gobject-introspection to find geany functions at runtime (through g_module_symbol()) in the future.
Agreed, but on the fact that it's not actually "also required for GI", it's *only* required for this kind of thing. It's good and everything, but not required but for dynamically looking up the functions.
- new API functions for registering keybindings (PR#376). The current
API functions do not allow context information to be stored and passed to the key handler function. This makes it very hard for non-C plugins to use these function. So what's needed are key handlers that get the necessary context information. This allows interprepted language plugins to register keybindings.
Agreed.
- A new plugin loader mechanism (a thread about this is already running
on the devel list): Similarly to the keybindings, the plugin_* functions implemented by plugins do not carry any context information, making it hard for non-C plugins to implement them properly. Therefore a new loader mechaism is needed so that the context information can be passed. The loader works such that an API function is called to register a function pointer table. This is crucial to possibly support plugins that register other plugins (so called pluxies) which is simply not possible with the current mechaism. The current loader is kept for backwards compatibility (but will not receive new features).
Agreed.
- New API functions to allow plugins to act as proxy plugins (pluxies).
These pluxies can then implement whatever is needed to execute code in the in the actual plugin, like invoking an interpreter or firing up a java vm. The pluxies call the new loader's API function on behalf of the actual plugin. The API function to implement the pluxies is a simple geany_register_pluxy() that, much like the normal plugin loader, that pluxies use to pass a function pointer table which implements the necessary hooks (probe(), load() and unload())
That's the part I'm really fuzzy about. I really don't see why we need this specific layer (maybe in an ideal world not bothering about how Geany currently does it): as asked on the other thread, why do we need anything beside geany_plugin_register() (required for everyone) and geany_plugin_unregister() (required only when registered sub-plugins, as the parent need to clean them up when it itself quits)?
Regards, Colomban
PS: damn, answered only on the user list.
Am 29.03.2015 um 19:17 schrieb Colomban Wendling:
Le 29/03/2015 00:23, Thomas Martitz a écrit :
- New API functions to allow plugins to act as proxy plugins (pluxies).
These pluxies can then implement whatever is needed to execute code in the in the actual plugin, like invoking an interpreter or firing up a java vm. The pluxies call the new loader's API function on behalf of the actual plugin. The API function to implement the pluxies is a simple geany_register_pluxy() that, much like the normal plugin loader, that pluxies use to pass a function pointer table which implements the necessary hooks (probe(), load() and unload())
That's the part I'm really fuzzy about. I really don't see why we need this specific layer (maybe in an ideal world not bothering about how Geany currently does it): as asked on the other thread, why do we need anything beside geany_plugin_register() (required for everyone) and geany_plugin_unregister() (required only when registered sub-plugins, as the parent need to clean them up when it itself quits)?
I'll answer this question here in the hope the other thread can concentrate on the new loader itself. But it also kind of depends on how I designed the loader so I guess there will be some cross talk.
As with git master, Geany's core loader scans the plugin folder on startup and on opening the PM dialog. For each recognized plugin file it allocates a GeanyPluginPrivate and calls geany_load_module().
So this is where we come from: - Geany initiates the scan (on behalf of the user) - Geany has certain files it recognizes as plugins. It won't attempt other files. - Geany allocates and frees the structure that holds everything it needs to know, this structure is opaque for actual plugins (and we need this to be able to extend it)
So, to support pluxies as first class citizens, I designed it such that Geany becomes able to scan additional files and hand the initialization of plugins to the proxy (instead of calling geany_load_module()). The idea is that Geany still performs the actual scan and builds up the plugin list in the PM dialog, so it's completely synchronous. Geany should still allocate the GeanyPluginPrivate structure for each plugin, for the proxied ones too. Note that proxy plugins too call the same geany_plugin_register() on behalf of the proxied plugins, like standard plugins have to too. Geany fully supervises which plugins are loaded, and when, so that it can provide a unified experience for the user, regardless of the plugin type that's behind the scenes.
Therefore my mechanism is based on plugins letting Geany know which files they can support, so that Geany's loader can scan for plugin files like ever, except it can match more types, and calls into the proxy to complete loading. For unloading the proxy is also called, to complete plugin finalization. For this, plugins need a way to provide Geany with the information it needs to load the additional plugins: file extension(s), and load/unload callbacks (and a probe, to resolve ambiguous file extensions).
To give a practical example:
with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
user opens PM dialog
1 Geany calls load_all_plugins(), which calls load_plugins_from_path($path) 2 for each $file in $path, Geany checks if the extension is G_MODULE_SUFFIX, and calls plugin_new($file, ...) 3 plugin_new() calls the plugins's geany_load_module() (if new-style plugin, calls version_check, set_info() for old-style ones) 4 geany_load_module() is implemented by the plugin, and registers itself with geany_plugin_register() < geany_plugin_register() adds the plugin to the plugin list, so that the PM can sort and show it
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file) which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins) 3* plugin_new() calls the load() hook registered by pluxies for the given extension. for standard plugins (without proxy) there is a predefined plugin_load_so() funtion that gets called instead. 4* The load-hook calls geany_plugin_register(), here Geany core and proxies work the same way
I designed it such, that the difference between standard plugins and proxied plugins is all contained in the load hook. The rest of Geany does not know about the difference. This ensures proxied plugins are first class citizens.
I hope you better understand my concept now. Let me emphasize again that it's focused having relatively little impact on Geany's core or existing plugins so that we can smoothly transition to a new non-C plugin world full of joy!
Best regards
[...]
Ok, this explains some of what I was asking on the other thread, so now I can ask the more specific questions below that are the key points in the confusion.
with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
user opens PM dialog
1 Geany calls load_all_plugins(), which calls load_plugins_from_path($path) 2 for each $file in $path, Geany checks if the extension is G_MODULE_SUFFIX, and calls plugin_new($file, ...) 3 plugin_new() calls the plugins's geany_load_module() (if new-style plugin, calls version_check, set_info() for old-style ones) 4 geany_load_module() is implemented by the plugin, and registers itself with geany_plugin_register() < geany_plugin_register() adds the plugin to the plugin list, so that the PM can sort and show it
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file)
What is is_plugin()? If its a function in Geany how does it get to know about new types of plugins without being hard coded?
which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
I'm guessing probe() is a function that looks for something in the .so that distinguishes if its new or old loader, but what about others?
3* plugin_new() calls the load() hook registered by pluxies for the given extension. for standard plugins (without proxy) there is a predefined plugin_load_so() funtion that gets called instead.
How does the load hook get defined for new types of plugins?
4* The load-hook calls geany_plugin_register(), here Geany core and proxies work the same way
Where is the geany_plugin_register() defined for a plugin written in a language that isn't C/C++/Vala that can produce a .so file?
I designed it such, that the difference between standard plugins and proxied plugins is all contained in the load hook. The rest of Geany does not know about the difference. This ensures proxied plugins are first class citizens.
Thats the correct target I agree, I just don't understand the design details yet.
Cheers Lex
Hello,
some if your question are easier to answer by looking at the code, I'll link the appropriate sections.
Am 30.03.2015 um 13:16 schrieb Lex Trotman:
[...]
Ok, this explains some of what I was asking on the other thread, so now I can ask the more specific questions below that are the key points in the confusion.
with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
user opens PM dialog
1 Geany calls load_all_plugins(), which calls load_plugins_from_path($path) 2 for each $file in $path, Geany checks if the extension is G_MODULE_SUFFIX, and calls plugin_new($file, ...) 3 plugin_new() calls the plugins's geany_load_module() (if new-style plugin, calls version_check, set_info() for old-style ones) 4 geany_load_module() is implemented by the plugin, and registers itself with geany_plugin_register() < geany_plugin_register() adds the plugin to the plugin list, so that the PM can sort and show it
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file)
What is is_plugin()? If its a function in Geany how does it get to know about new types of plugins without being hard coded?
It's new, small helper function I added. It loops through all known file extensions, and returns the first pluxy (a PluginProxy *) for which a) the supported file extension matches and b) the probe hook returned true (or is NULL, for standard plugins). $file is not a plugin if is_plugin returns NULL, i.e. no pluxy was found.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L844
File extensions and the proxy hooks (probe, load, unload) are registered by a plugin during its init() through the new plugin_register_proxy() function. Here the pluxy added to the list of registered pluxies. This list is initialized with the simulated pluxy that provides standard plugins (this is not a plugin, it's contained in plugins.c, it's just to keep the code paths equal).
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
I'm guessing probe() is a function that looks for something in the .so that distinguishes if its new or old loader, but what about others?
It depends on the pluxy what the prope() function does! For my peasy pluxy (that provides generic support for libpeas-based plugins), it looks if there is a matching *.plugin for a given *.so, and if yes return a code so that Geany does not attempt to process the .so itself.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L73
There is no probe() for standard plugins, it accepts all .so. Whether it's a new or old style plugin is determined later. It *could* be in a probe() hook for standard plugins as well, I just didn't happen to implement it that way (yet), because plugin_load_so needs to distinguish between the two anyway.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L472
3* plugin_new() calls the load() hook registered by pluxies for the given extension. for standard plugins (without proxy) there is a predefined plugin_load_so() funtion that gets called instead.
How does the load hook get defined for new types of plugins?
Via the new API function plugin_register_proxy(). https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
4* The load-hook calls geany_plugin_register(), here Geany core and proxies work the same way
Where is the geany_plugin_register() defined for a plugin written in a language that isn't C/C++/Vala that can produce a .so file?
In the load hook of the pluxy. Either the pluxy calls it directly or it decides to provide a suitable binding so that the non-C script can call it itself, but it has to be during the execution of the load hook.
The demopluxy does it in the load() hook, right after parsing the metadata of the example plugin (I completely made up a fake plugin format for demonstration purposes):
https://github.com/kugel-/geany/blob/pluxy/plugins/demopluxy.c#L169
I have done it the same way for peasy too, because libpeas plugins can be python or js, and I didn't create bindings yet. But it shows working this way.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L130
I designed it such, that the difference between standard plugins and proxied plugins is all contained in the load hook. The rest of Geany does not know about the difference. This ensures proxied plugins are first class citizens.
Thats the correct target I agree, I just don't understand the design details yet.
Thanks for the heads up!
Best regards
On 30 March 2015 at 22:43, Thomas Martitz kugel@rockbox.org wrote:
Hello,
some if your question are easier to answer by looking at the code, I'll link the appropriate sections.
No problem, the explanations below about how things are meant to be used are more help than looking at the code by itself, what is needed is the user guide, which these explanations make a start at, don't lose them :)
Am 30.03.2015 um 13:16 schrieb Lex Trotman:
[...]
Ok, this explains some of what I was asking on the other thread, so now I can ask the more specific questions below that are the key points in the confusion.
with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
user opens PM dialog
1 Geany calls load_all_plugins(), which calls load_plugins_from_path($path) 2 for each $file in $path, Geany checks if the extension is G_MODULE_SUFFIX, and calls plugin_new($file, ...) 3 plugin_new() calls the plugins's geany_load_module() (if new-style plugin, calls version_check, set_info() for old-style ones) 4 geany_load_module() is implemented by the plugin, and registers itself with geany_plugin_register() < geany_plugin_register() adds the plugin to the plugin list, so that the PM can sort and show it
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file)
What is is_plugin()? If its a function in Geany how does it get to know about new types of plugins without being hard coded?
It knows because the extensions are registered by the function below called by another plugin right? How do you make sure that the register function has been called before you come across a file with that extension?
It's new, small helper function I added. It loops through all known file extensions, and returns the first pluxy (a PluginProxy *) for which a) the supported file extension matches and b) the probe hook returned true (or is NULL, for standard plugins). $file is not a plugin if is_plugin returns NULL, i.e. no pluxy was found.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L844
File extensions and the proxy hooks (probe, load, unload) are registered by a plugin during its init() through the new plugin_register_proxy() function.
Ok, so this registers a new type of plugin by its extension(s) and that type is associated with a plugin that provides:
- the loader functionality? - interface wrappers/bindings (like geanypy does)? - starts/loads any other things, like the Python interpretor or a JVM or Haskell runtime?
Here the pluxy added to the list of registered pluxies. This list is initialized with the simulated pluxy that provides standard plugins (this is not a plugin, it's contained in plugins.c, it's just to keep the code paths equal).
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
I'm guessing probe() is a function that looks for something in the .so that distinguishes if its new or old loader, but what about others?
It depends on the pluxy what the prope() function does! For my peasy pluxy (that provides generic support for libpeas-based plugins), it looks if there is a matching *.plugin for a given *.so, and if yes return a code so that Geany does not attempt to process the .so itself.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L73
There is no probe() for standard plugins, it accepts all .so.
If it doesn't call probe how does it know if its a traditional plugin, or a peas one or maybe some other version that makes a .so file? (Haskell anybody :)
Whether it's a new or old style plugin is determined later. It *could* be in a probe() hook for standard plugins as well, I just didn't happen to implement it that way (yet), because plugin_load_so needs to distinguish between the two anyway.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L472
3* plugin_new() calls the load() hook registered by pluxies for the given extension. for standard plugins (without proxy) there is a predefined plugin_load_so() funtion that gets called instead.
How does the load hook get defined for new types of plugins?
Via the new API function plugin_register_proxy(). https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
Ok, understand now
4* The load-hook calls geany_plugin_register(), here Geany core and proxies work the same way
Where is the geany_plugin_register() defined for a plugin written in a language that isn't C/C++/Vala that can produce a .so file?
In the load hook of the pluxy. Either the pluxy calls it directly or it decides to provide a suitable binding so that the non-C script can call it itself, but it has to be during the execution of the load hook.
Ok, so if there are all these hooks, probably later the traditional plugins can be just another pre-registered set of hooks built into Geany, but I agree with your approach of not trying to do that all in the first step, leave the existing code as little changed as possible until the new system settles down and then change the existing code to use it.
The demopluxy does it in the load() hook, right after parsing the metadata of the example plugin (I completely made up a fake plugin format for demonstration purposes):
https://github.com/kugel-/geany/blob/pluxy/plugins/demopluxy.c#L169
I have done it the same way for peasy too, because libpeas plugins can be python or js, and I didn't create bindings yet. But it shows working this way.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L130
I designed it such, that the difference between standard plugins and proxied plugins is all contained in the load hook. The rest of Geany does not know about the difference. This ensures proxied plugins are first class citizens.
Thats the correct target I agree, I just don't understand the design details yet.
Thanks for the heads up!
Best regards _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 30.03.2015 um 14:33 schrieb Lex Trotman:
What is is_plugin()? If its a function in Geany how does it get to know about new types of plugins without being hard coded?
It knows because the extensions are registered by the function below called by another plugin right? How do you make sure that the register function has been called before you come across a file with that extension?
For now, I kept it simple: Geany simply restarts the "scan for plugins" loop when new extensions are added during the the scan (remember that during the scan, each file is loaded and its init() is called, before the next file is even attempted). The PM dialog is refreshed in the same way when a pluxy is activated by the user.
It can be made smarter, but it's good enough at the moment.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L878 and https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1272
It's new, small helper function I added. It loops through all known file extensions, and returns the first pluxy (a PluginProxy *) for which a) the supported file extension matches and b) the probe hook returned true (or is NULL, for standard plugins). $file is not a plugin if is_plugin returns NULL, i.e. no pluxy was found.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L844
File extensions and the proxy hooks (probe, load, unload) are registered by a plugin during its init() through the new plugin_register_proxy() function.
Ok, so this registers a new type of plugin by its extension(s) and that type is associated with a plugin that provides:
- the loader functionality?
- interface wrappers/bindings (like geanypy does)?
- starts/loads any other things, like the Python interpretor or a JVM
or Haskell runtime?
Yes, but the 2nd and 3rd points are entirely up to the pluxy. My demopluxy.so doesn't do anything fancy. it creates a dummy plugin out of a GKeyFile.
But the loader/unloader function is mandatory, it also acts as the entry point for pluxies to start their bindings/vm machinery if necessary.
Here the pluxy added to the list of registered pluxies. This list is initialized with the simulated pluxy that provides standard plugins (this is not a plugin, it's contained in plugins.c, it's just to keep the code paths equal).
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
I'm guessing probe() is a function that looks for something in the .so that distinguishes if its new or old loader, but what about others?
It depends on the pluxy what the prope() function does! For my peasy pluxy (that provides generic support for libpeas-based plugins), it looks if there is a matching *.plugin for a given *.so, and if yes return a code so that Geany does not attempt to process the .so itself.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L73
There is no probe() for standard plugins, it accepts all .so.
If it doesn't call probe how does it know if its a traditional plugin, or a peas one or maybe some other version that makes a .so file? (Haskell anybody :)
Geany doesn't and cannot know it's a peas plugin, though it could be smarter at determining it's a standard plugin. But I currently implemented a scheme where the first pluxy that matches wins, but if it doesn't match all other pluxes are tried. Geany itself is tried last.
This should work for all real-world cases, even when there are multiple (more than 2) providers for .so files. If the pluxies are accurate enough at determining their own filetypes then no conflicts arise.
Geany is always tried last, so if it gets to process a .so file, then it assumes it's a standard plugin like it's done in git master (no change here)
In the load hook of the pluxy. Either the pluxy calls it directly or it decides to provide a suitable binding so that the non-C script can call it itself, but it has to be during the execution of the load hook.
Ok, so if there are all these hooks, probably later the traditional plugins can be just another pre-registered set of hooks built into Geany, but I agree with your approach of not trying to do that all in the first step, leave the existing code as little changed as possible until the new system settles down and then change the existing code to use it.
It's done this way already, the hooks are compiled into Geany. As I said the list of pluxies is initialized with a simulated one that provides the standard plugins { .extension = { "so", NULL }, .probe = NULL, .load = plugin_load_so, .unload = plugin_unload_so };
Best regards
Le 30/03/2015 00:17, Thomas Martitz a écrit :
Am 29.03.2015 um 19:17 schrieb Colomban Wendling:
Le 29/03/2015 00:23, Thomas Martitz a écrit :
- New API functions to allow plugins to act as proxy plugins (pluxies).
[…]
That's the part I'm really fuzzy about. I really don't see why we need this specific layer […]
[…]
As with git master, Geany's core loader scans the plugin folder on startup and on opening the PM dialog. For each recognized plugin file it allocates a GeanyPluginPrivate and calls geany_load_module().
[…] with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
[…]
OK, fair enough indeed. And well, proxy plugins are special enough to warrant their own API if it's useful anyway, so okay.
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file) which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
As raised on IRC, one small question: do we need a file extension if we have probe()? I don't mind much, but I would imagine probe() could filter extensions itself and simply return the appropriate value. This would also potentially allow for extensionless plugins.
But that's a small detail, and apart feeling it a little redundant I don't mind either way.
I hope you better understand my concept now. […]
Yep, I do, thanks for these very good clarifications :)
Regards, Colomban
On 30 March 2015 at 23:54, Thomas Martitz kugel@rockbox.org wrote:
Am 30.03.2015 um 14:33 schrieb Lex Trotman:
What is is_plugin()? If its a function in Geany how does it get to know about new types of plugins without being hard coded?
It knows because the extensions are registered by the function below called by another plugin right? How do you make sure that the register function has been called before you come across a file with that extension?
For now, I kept it simple: Geany simply restarts the "scan for plugins" loop when new extensions are added during the the scan (remember that during the scan, each file is loaded and its init() is called, before the next file is even attempted). The PM dialog is refreshed in the same way when a pluxy is activated by the user.
It can be made smarter, but it's good enough at the moment.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L878 and https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1272
Ok, so long as it doesn't re-load plugins already loaded thats ok, the PM might get a bit slow, but its not something thats used every minute of every day.
It's new, small helper function I added. It loops through all known file extensions, and returns the first pluxy (a PluginProxy *) for which a) the supported file extension matches and b) the probe hook returned true (or is NULL, for standard plugins). $file is not a plugin if is_plugin returns NULL, i.e. no pluxy was found.
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L844
File extensions and the proxy hooks (probe, load, unload) are registered by a plugin during its init() through the new plugin_register_proxy() function.
Ok, so this registers a new type of plugin by its extension(s) and that type is associated with a plugin that provides:
- the loader functionality?
- interface wrappers/bindings (like geanypy does)?
- starts/loads any other things, like the Python interpretor or a JVM
or Haskell runtime?
Yes, but the 2nd and 3rd points are entirely up to the pluxy. My demopluxy.so doesn't do anything fancy. it creates a dummy plugin out of a GKeyFile.
Oh sure it depends on the language/runtime etc.
But the loader/unloader function is mandatory, it also acts as the entry point for pluxies to start their bindings/vm machinery if necessary.
Thats the load for the proxy right? The load for the supported plugins is one of the registered hooks right?
That means that the "machinery" gets started even if there are no plugins using it enabled?
Here the pluxy added to the list of registered pluxies. This list is initialized with the simulated pluxy that provides standard plugins (this is not a plugin, it's contained in plugins.c, it's just to keep the code paths equal).
https://github.com/kugel-/geany/blob/pluxy/src/plugins.c#L1601
which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
I'm guessing probe() is a function that looks for something in the .so that distinguishes if its new or old loader, but what about others?
It depends on the pluxy what the prope() function does! For my peasy pluxy (that provides generic support for libpeas-based plugins), it looks if there is a matching *.plugin for a given *.so, and if yes return a code so that Geany does not attempt to process the .so itself.
https://github.com/kugel-/peasy/blob/master/src/peasy.c#L73
There is no probe() for standard plugins, it accepts all .so.
If it doesn't call probe how does it know if its a traditional plugin, or a peas one or maybe some other version that makes a .so file? (Haskell anybody :)
Geany doesn't and cannot know it's a peas plugin, though it could be smarter at determining it's a standard plugin. But I currently implemented a scheme where the first pluxy that matches wins, but if it doesn't match all other pluxes are tried. Geany itself is tried last.
This should work for all real-world cases, even when there are multiple (more than 2) providers for .so files. If the pluxies are accurate enough at determining their own filetypes then no conflicts arise.
Geany is always tried last, so if it gets to process a .so file, then it assumes it's a standard plugin like it's done in git master (no change here)
Ok
In the load hook of the pluxy. Either the pluxy calls it directly or it decides to provide a suitable binding so that the non-C script can call it itself, but it has to be during the execution of the load hook.
Ok, so if there are all these hooks, probably later the traditional plugins can be just another pre-registered set of hooks built into Geany, but I agree with your approach of not trying to do that all in the first step, leave the existing code as little changed as possible until the new system settles down and then change the existing code to use it.
It's done this way already, the hooks are compiled into Geany. As I said the list of pluxies is initialized with a simulated one that provides the standard plugins { .extension = { "so", NULL }, .probe = NULL, .load = plugin_load_so, .unload = plugin_unload_so };
Ok, missed that, fine.
Cheers Lex
Best regards _______________________________________________ Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 31.03.2015 um 01:48 schrieb Lex Trotman:
Yes, but the 2nd and 3rd points are entirely up to the pluxy. My demopluxy.so doesn't do anything fancy. it creates a dummy plugin out of a GKeyFile. Oh sure it depends on the language/runtime etc.
But the loader/unloader function is mandatory, it also acts as the entry point for pluxies to start their bindings/vm machinery if necessary.
Thats the load for the proxy right? The load for the supported plugins is one of the registered hooks right?
That means that the "machinery" gets started even if there are no plugins using it enabled?
Again, how/when it loads the machinery depends on the proxy :) But the load/unload I was referring to is called per-sub-plugin, so a smart proxy can start the machinery (if there is anything to start) when the first plugin is loaded, and /not/ in the proxy's own init().
Best regards.
On 31 March 2015 at 16:51, Thomas Martitz kugel@rockbox.org wrote:
Am 31.03.2015 um 01:48 schrieb Lex Trotman:
Yes, but the 2nd and 3rd points are entirely up to the pluxy. My demopluxy.so doesn't do anything fancy. it creates a dummy plugin out of a GKeyFile. Oh sure it depends on the language/runtime etc.
But the loader/unloader function is mandatory, it also acts as the entry point for pluxies to start their bindings/vm machinery if necessary.
Thats the load for the proxy right? The load for the supported plugins is one of the registered hooks right?
That means that the "machinery" gets started even if there are no plugins using it enabled?
Again, how/when it loads the machinery depends on the proxy :) But the load/unload I was referring to is called per-sub-plugin, so a smart proxy can start the machinery (if there is anything to start) when the first plugin is loaded, and /not/ in the proxy's own init().
Neat, ok :)
Best regards.
Devel mailing list Devel@lists.geany.org https://lists.geany.org/cgi-bin/mailman/listinfo/devel
Am 30.03.2015 um 15:48 schrieb Colomban Wendling:
Le 30/03/2015 00:17, Thomas Martitz a écrit :
Am 29.03.2015 um 19:17 schrieb Colomban Wendling:
Le 29/03/2015 00:23, Thomas Martitz a écrit :
- New API functions to allow plugins to act as proxy plugins (pluxies).
[…]
That's the part I'm really fuzzy about. I really don't see why we need this specific layer […]
[…]
As with git master, Geany's core loader scans the plugin folder on startup and on opening the PM dialog. For each recognized plugin file it allocates a GeanyPluginPrivate and calls geany_load_module().
[…] with my new loader (no pluxies) it goes like this, and this is *very* similar to git master.
[…]
OK, fair enough indeed. And well, proxy plugins are special enough to warrant their own API if it's useful anyway, so okay.
Yea, since the plan is to integrate the sub-plugins in the same UI as standard plugins, *some* kind of API is needed anyway. And I think my approach gives great flexibility to the pluxies while maintaining Geany (and the user) as the "supervisor". And this by only adding a single API function.
Now, with pluxies, it is completely the same except for: 2* for each $file in $path, Geany calls is_plugin($file) which matches additional file extensions (as provided by pluxies), it also calls the probe() hook to resolve ambiguous files (e.g. .so files, they can be core or libpeas plugins)
As raised on IRC, one small question: do we need a file extension if we have probe()? I don't mind much, but I would imagine probe() could filter extensions itself and simply return the appropriate value. This would also potentially allow for extensionless plugins.
But that's a small detail, and apart feeling it a little redundant I don't mind either way.
I think it's useful for a number of reasons: - It's less duplicated code in the pluxies because virtually all of them need to have one or more file extension checks (even if it's just less than 10 lines) - We could potentially give the user a more unified UI that sums up loaded pluxies - Knowing the file extension in the core could become handy in the feature, like if we ever want to auto-activate pluxies for certain extensions (e.g. for pluxies that we ship with Geany itself), or reporting conflicting pluxies to the user. We wouldn't have to introduce a new API then - Although very minor: we can safe the indirect function call into the pluxy if we already know the negative result
We could support extension-less sub-plugins either way (e.g. by treating the ""/NUL extension specially) so we're not limiting ourself. However it's true that it's not *strictly* needed.
I hope you better understand my concept now. […]
Yep, I do, thanks for these very good clarifications :)
You're welcome :)
Best regards