The regular expression engine used by Geany is one that is included in the Glib library it already uses, so no additional dependencies are needed. Therefore it is unlikely that an alternative engine will be used.

Also the regex engine replace cannot be used as the change is not to a plain string, it is to an edit buffer and has to be made with the modification functions that the edit widget provides.

There may be some available options of Glib regex that are not exposed by Geany which could be added if there is a use-case.