You know what? I actually changed my mind thinking about this lately, and now agree the best way to go is probably just use whatever upstream does. The goal is to use upstream as if it was a library, and actually target making it one that we could simply use. And one of the prominent CTags features is custom parser definitions (even more so with recent uctags additions), and even though we don't support this in Geany yet, it probably shouldn't be stripped from a libuctags.

So, I'll withdraw my "please, no duplicated regex libraries" argument. I'm still not in love with it, but now think it's probably the lesser of the evils (esp. if we can use the OS' regex one in most setups -- but the odd Windows kid). I have no idea how easy it'll be to get the regex parsing code in, though.

Whats involved in "getting the regex parsing code in"? Isn't it all inside the cobol.c?

I'm not sure what's involved exactly, but no, it's not all in cobol.c otherwise it wouldn't have been a problem to just keep it :) The thing is that it needs the whole regex infrastructure from uctags, which we currently don't have. So that's another bunch of stuff to fetch and adapt from there.

On actionscript, I build uctags from git this afternoon, and `ctags --list-languages` doesn't list it, so it would appear it has been dropped?

@b4n @techee ping, any comment on using upstream flex as the actionscript parser?

Umm, hello, is nobody going to answer the questions above about actionscript parser?

@b4n Ahh ha, so if that happens and my suggestion that flex.c is an actionscript parser then we won't need a regex infrastructure anyway :)

@elextr If only I could try Geany master behavior and compare results… :grin:

@b4n @techee ping, any comment on using upstream flex as the actionscript parser?

A quick test (not thorough at all) seems to show that yes, upstream has `flex` for ActionScript, but also that is very limited and worse than what we had. I'll look a little deeper into this, but it kinda sounds like I'll have to learn another language…

@b4n, can you improve the parser thats part of flex.c rather than starting a new one from scratch?

Some pain later: https://github.com/universal-ctags/ctags/pull/2084 (actually, once I gave up importing separate commits, it became very tedious but less painful).

@masatake See https://github.com/geany/geany/pull/2132 - I've added back support for regex parsers.

I guess libctags in geany may disable some unwanted features. It is understandable to me. Some features I have added to u-ctags is so immature. u-ctags should provide the way to disable such features at build-time or run-time.

Could you write a short document about libctags internal? Based on the document, I will try to move libctags code from Geany to u-ctags incrementally.

Could you write a short document about libctags internal?

There's not much to write. Basically we just need to:

* bypass some things from ctags main() which are irrelevant for us (command-line options etc.) - this is done in ctagsInit() inside ctags.api.c * register our custom writer (done inside ctagsInit()) which unlike other writers doesn't write to the MIO (which is NULL in our case) but instead the writeEntry() passes the tag information to Geany * be informed when re-parsing happens so we can invalidate the tags we passed to Geany - this is currently done in a hacky way but it could be nicer if there was a function for it in a writer * be able to initiate parsing by ourselves - either from a file or from a char buffer - see ctagsParse() in ctags-api.c * get some basic information about the registered parsers like kinds they provide etc. - see the rest of ctags-api.c

This is basically the main functionality, there are some other diffs where we for instance don't have fnmatch.h or where the parser list order has to correspond to the order specified in Geany which we could maybe change in Geany itself. But that's basically it.

@techee @masatake @b4n

I assume you are talking about having u-ctags provide a library that other apps can use? Thats a good idea. Why not refactor u-ctags to separate the parsers and parsing infrastructure from the command line and file writing stuff. That way such a library only includes parsing, thats the part most apps want and the library can be used by the u-ctags app of course :).

As I think @techee proposed apps (including u-ctags itself) should register functions to provide things like reading files and accepting tags, for u-ctags to write to disk (I presume) and for Geany to add to symbols.

Also apps need to be able to not build in parsers for languages they don't want, there are twice as many files in u-ctags/parsers as in Geany/parsers directory.

I'm very surprised that @b4n has rewritten two parsers of u-ctags. Improving Cobol and Flex parsers are welcome but... @b4n, what will you do if someone adds more regex basedd parsers to u-ctags. Replacing the regex engine of u-ctags is more effortless.

Adding support for u-ctags regex parser would be better and is the goal, but it's not easy, and certainly not effortless -- see @techee's work :slightly_smiling_face: The reason why I worked on the parsers is that we'd need something working in the next release for COBOL and ActionScript, relase which is supposed to happen next weekend, which is awful close, and I didn't expect getting the regex parsers to work would be a trivial task that would take a couple of hours and be robust. So I figured that it was safer to work on 2 parsers, as if there really was a problem with them it would only affect users of those, not potentially everyone. And well, while at it I figured I could probably improve parsing for the relevant languages given those regex parsers seemed quite basic. Also to be fair you failed to take into account my tendency to write parsers for languages I don't know :grinning:

--- I'll answer the rest later, hopefully today.

Why not refactor u-ctags to separate the parsers and parsing infrastructure from the command line and file writing stuff. That way such a library only includes parsing, thats the part most apps want and the library can be used by the u-ctags app of course :).

Yes, that would be the ideal long-term solution. And actually the writing stuff should probably be part of such a library too because some applications may want to write the tags files - it's just the command-line interface that should be separate. But doing this right means a lot of thinking about what the official API should look like and in the end may need a lot of refactoring so first I'd just try to bypass what we don't want to happen for Geany like the command-line interface add some missing functionality we need in Geany (and there's very little of new code, it's just something like 100 lines) and use the internal ctags API for now. I could create some dummy client application for uctags which uses the same functionality we need in Geany to make sure some internal change in uctags doesn't remove or break what we need.

Also apps need to be able to not build in parsers for languages they don't want, there are twice as many files in u-ctags/parsers as in Geany/parsers directory.

Yeah, but in this case it's our business to remove the parsers we don't want to get compiled like we do with Scintilla:

https://github.com/geany/geany/blob/master/scintilla/scintilla_changes.patch

@elextr,

If it was a proper library then both Geany and ctags and others should be able to use it as a library. There should be no need to import ctags code into Geany at all if it was a proper library, just link to the API.

This is ideal but I would like to focus on the interface between Geany. I don't have enough skill to design and maintain a new library API. 3 ~ 5 years ago, I declined to make ctags a library in the above reason. However, libctags is already there. I don't have to design an API. I think I can do keeping the interface. The benefit of moving the library to u-ctags project is that I, an active ctags developer, can take care of keeping the interface working.

The following figure shows the position of libctags in us.


For a while, the libctags is only for Geany. I assume Geany may incorporate ctags code via git subtree.

@techee

I could create some dummy client application for uctags which uses the same functionality we need in Geany to make sure some internal change in uctags doesn't remove or break what we need.

YES! It is what we need. Let's give an impressive name the dummy client. This client drives us. My idea is 'minigeany' or 'microgeany'. The client is at uctags side so we can add test cases on it.

However, libctags is already there.

@masatake I assume you mean the libctags that Geany builds?

@b4n @techee, does that have an API? My understanding is that it is just a collection of pieces of u-ctags for use by Geany? And why do we build it as a library anyway, why not just link it?

This is ideal but I would like to focus on the interface between Geany.

I don't have enough skill to design and maintain a new library API. 3 ~ 5 years ago, I declined to make ctags a library in the above reason. However, libctags is already there. I don't have to design an API. I think I can do keeping the interface. The benefit of moving the library to u-ctags project is that I, an active ctags developer, can take care of keeping the interface working.

Totally agree, I think uctags shouldn't go crazy about making it a library - it's primarily a command-line program and with Geany being the only application (mis)using it as a library, there's no need to spend much effort on it. For us it's not really a problem if some internal interface changes - we can fix the code in Geany easily as long as the core required functionality is accessible to us in some way. The main purpose of the ctags-api.c/h file (which I moved back to tag manager in the latest version) is to isolate Geany from the rest of ctags so this is the only file we'd have to update if something changes.

YES! It is what we need. Let's give an impressive name the dummy client. This client drives us.

My idea is 'minigeany' or 'microgeany'. The client is at uctags side so we can add test cases on it.

Good, I guess I'll do it now. The weather is beautiful outside and it would be really stupid not to sit inside and code ;-). I've made some more changes in #2132 to improve how we interface with ctags and creating such a client should be really easy now. I'll open a pull request in uctags when I'm done.

See #2134 for ActionScript.

Closed #2119.