Hi Geany Devs,
I am having a bit of a problem with Geany 1.30.1, I use Geany on Windows and Linux and I have not been able to track down the issue.
It started happening for one file around June 2017 after many years of using geany with no issue's.
The file refuses to accept any typing or copying, after sometimes three, four or in the case of the attached jpg seven letters the file locks up and either adds LF or CRLF or some other chars over the top of the normal text appear, its then totally locked until I restart Geany.
I have been searching for the cause without coming to a solution, some time back in June when it started happening it kind of went away for a week or two now it happens constantly every time I open the file.
- At first I thought at first it was a line ending issue or some kind if it is I am not sure what I can do to fix the issue.
- I have tried changing the encoding of the file to see if this helped but so far it does it on every type of encoding I have tried (using UTF-8).
- I also thought it might be an illegal character, or a character that geany does not like ((( but searching for a character in such a large file seems problematic ))). Does Geany give out any errors ?
- lots of things go through my mind on what the cause could be but over time I have tested the geany and believed that it can / could handle larger files without issues. So I am hoping its not the size of the file that is now too big for Geany.
Required Infomation : The version number of Geany 1.30.1
The version number of GTK+ (can be printed with geany -V) Geany: Geany 1.30.1, English_United Kingdom.1252 Geany: GTK 2.24.31, GLib 2.50.3 Geany: System data dir: C:\Program Files (x86)\Geany\data Geany: User config dir: C:\Users\darren.foster\AppData\Roaming\geany Geany: System plugin path: C:\Program Files (x86)\Geany\lib\geany Geany: Added filetype Arduino (61). Geany: Added filetype Clojure (62). Geany: Added filetype CUDA (63). Geany: Added filetype Cython (64). Geany: Added filetype Genie (65). Geany: Added filetype Graphviz (66). Geany: Added filetype JSON (67). Geany: Added filetype Scala (68). Geany: unknown : None (UTF-8)
I am hopeful that you have encountered this issue and there is a simple solution which means I can continue using the file as it has been a very useful up until now, .
If there is any more information I can provide please do let me know from where.
Help please thanks Daz
The image above looks weird, the LF characters should be on the ends of the lines, not the beginning.
Hi Elextra,
Here's what happens on copy n paste :
I don't really have a logical solution to track down where the issue might be a lift and shift would probably take months...
Cheers Daz
You said Linux and Windows, is the problem on both?
The last screenshot shows you have mixed line endings, the screen shows CRLFs but Geany is in LF mode?
But having CR before the LFs should not stop the line ends being recognised.
What if you do `Document->Set Line Endings->Convert and set to <whatever is NOT selected>` to make the document consistent.
Also how big in bytes is your file?
I had a problem with a file not opening at all, it turned out that there was a NULL byte embedded in the file. Check the file for special characters, `cat -vet` may help if its not too big otherwise `grep`.
@AdamDanischewski Geany deliberately rejects files containing NUL characters because that stops all the C code that uses NUL terminated strings from working, but that is not the problem here, the file clearly opens.
Hi Thanks for you comments, I have not found the solution to the issue yet...
I did try a few things :
I tied to narrow down where in the document the issue might be still working on this one, and the issue does move to a new document at some point but my copying and pasting was a good few hundred thousands lines at a time.
had a look at non-ascii chars with grep --color='auto' -P -n '[^\x00-\x7F]'
this works out to be around 209560 lines and I am not sure if this will show null chars or not.
tried looking for null chars with grep -m 1 -LE "[^]" Info2add.txt
but nothing returned....
the file is almost 300 MB in size... so using cat / grep would only really help if I knew what to look for...
I'll keep looking as this is pretty important to me and my productivity is taking a massive hit....
Thanks for your comments, any ideas comments on how to stop this from happening in the future would be great to hear
To be honest I have not tested this on Linux since it started happening, I will and let you know. All I wanted to say was that normally I used the same file on box OSes.
Cheers Daz
`grep`'ing `'[^\x00-\x7F]'` will show you everything that is not in the ascii range. You are negating the set so it's not going to show you `\x00` that's null and represented in the ascii range. Is there are a reason there are so many non ascii characters in this file? If there is other code hitting this are they "polluting" it? Beyond learning how to fix it, you probably should find out why this is happening.
You could try deleting the null bytes, to see if that helps: ``` cat file | tr -d '\000' > new_file
diff file new_file ``` You can also try segmenting the file (for testing) into smaller chunks until you find the problem area. You may want to use `od` for a closer inspection.
@AdamDanischewski like I said above, Geany will NOT read files with NULs in them, no point in searching for them, also it validates UTF-8 inputs. Its fine with non-ASCII characters so long as they are valid UTF-8.
@hdaz000700 300Mb should be ok, so long as you have enough memory, but that would have other effects, not weird displays. The screen shots you posted above, are they at the start of the files or far down it?
@hdaz000700 can you try opening the file in Scite, I thin k its available for windows as well as Linux, its the reference platfor for Scintilla
after doing cat file | tr -d '\000' > new_file diff file new_file
diff shows nothing and the md5sums are the same.
reason there are so many non ascii characters There are lots of information things like ALT Characters http://www.tedmontgomery.com/tutorial/altchrc-a.html <<< been in the same file for years now >>>
lots of newer console output are using things like "●" docker, systemd, pip etc
It also appear some chars have corrupted to some extent £ has turned into "£" as an example.
Cheers Daz
It also appear some chars have corrupted to some extent £ has turned into "£" as an example.
That sounds like you may have pasted stuff with different encodings.
Have you tried segmenting the file? Copy it someplace and cut it in half, are both sides broken? If so keep cutting it in half again until you find a half not broken then focus on the smallest broken half continuing to narrow it down to the problem area.
Hi Adam,
Yes I have started working on segmentation but with that technique its going to take some time before, I track down where the issue is and there could be more than one....
I'll update if and when I do fine the issues... :)
Cheers Daz
It may seem daunting because of the amount of rows, but it shouldn't take very long since its a binary search the complexity is O(log n) at worst.
Hi Adam,
Thanks for alias, although I think I have hit on the issue and I don't believe it is a char issue or special char issue at all.
So after further testing I can 100% confirm it does not happen under linux but is happening on my setup for windows7. (( I don't believe its memory issue the laptop has 16GB and it happens even when nothing else is running i.e. tones of memory free ))
I can recreate this with using yes and random lines of "1" within the file to at least 265 MB, I set the file to 300MB on the test - So believe there is some limit I am hitting i.e. size of file or number of lines, words of chars..
Lines : 4517796 Words : 31143087 Characters : 267830878
It would be grand if someone else could test on windows as I don't have another windows laptop.
Regards Daz
@hdaz000700 correct, it is almost certainly nothing to do with character problems, your screen shots show the file was loaded as UTF-8 in which case its is validated and you would get an error if its wasn't valid UTF-8.
One of the differences between Linux and Windows is the size of some integer data types which are 64 bits on Linux and 32 bits on windows, but none of your numbers above come close to the 2147483647 limit of a signed 32 bit number. (BTW that limit means maximum file size of 2GB, so that would only use 4GB plus a bit, so your 16GB is largely unemployed :)
Since the main symptom is a scrambled display, a main suspicion is the Scintilla editing widget Geany uses or the GTK and Pango graphics and text display libraries that it uses. Being native Linux libraries GTK and Pango have much more testing there than the Windows ports do.
Scintilla has multiple backends, GTK, Windows and OSX but Geany uses the GTK one on all platforms. If you, or somebody who can reproduce the problem, could also try the Scite editor on Windows since it uses Scintilla but with the Windows backend it would possibly separate the problem into Scintilla or GTK/Pango.
PS Scite is available from http://www.scintilla.org/SciTEDownload.html
Hi elextr,
If I open in SciTE
Failure in Scintilla Memory exhausted. SciTE will now close
Regards Darren
What did you copy from? How big a file are you now using? I believe that Windows actually copies to the clipboard so that means at least twice the memory. What about just opening the file?
Scite for windows is 32 bit, @eht16 is Geany for windows 32 or 64 bit?
Geany for windows 32 or 64 bit
The official release is 32-bit.
Hi Elextr,
I copied from a file opened in geany <<< small file >>> about ten lines of text even with geany closed opening SciTE and copy n pasting from notepad the three lines from Windows Binaries below gives the same behaviour.
Hi Codebrainz, How would one know which version where in Geany would one find this information ?
https://www.geany.org/Download/Releases
Windows Binaries For Windows XP or newer. geany-1.31_setup.exe - Full Installer including GTK 2.24 runtime
Regards Darren
I am confused, your nitial report is about opening a file, now you are talking about copying and pasting?
To be clear opening a file and pasting information into an already open buffer are very different operations.
I am sorry if I was not clear but I believe I wrote "refuses to accept any typing or copying" after a certain amount of char's the file in geany locks up and shows the first screen shoot, if copy n paste is in play then it normally locks up in a similar pattern to the second screenshot .
My logic when typing the first report, was they issues are similar or the same root cause.
Doing the same operation within SciTE first just entering text does not seem to lock up but I would need to do more testing... with copy n paste it gives the error and CR LR random pattern across the file.
Either way where ever the issues may be it would be really grand to find a solution and move past them and not have them...
Regards Darren
At 300MB on a Linux box my Geany freezes up too. Have you considered breaking the file into smaller chunks for processing? E.g. The way syslog manages logs by creating sequenced files. If you need to keep that size and edit it every so often, I recommend you use an editor that can *NOT* load the whole file such as `vi` (hit Ctrl-C when it starts loading to cancel enumeration) - it's a powerful editor to learn anyway.
I can open files larger than 500MB just fine in Geany on my 4GB Linux laptop, (albiet with nothing else running, @AdamDanischewski close your browser :). It takes about 6 secs to open or to move the cursor to the end of the file. That seems ok for 1/2 GB file on a laptop. Scintilla uses two bytes of memory for each byte in the file, plus some for each line, so memory usage is at least double the file size and more is needed during opening to handle encoding conversions and UTF-8 validation. As soon as the file is big enough that memory usage reaches 4GB, it causes swapping and it looks like it froze, but in fact it will come back after a time, its just that swapping to disk is slooooow. Of course if you fill up swap all bets are off :)
@hdaz000700 Operations like copy and paste result in several copies in memory at once, reaching the limit of memory faster than the file itself does, and so starting swapping and so appearing to freeze sooner. So if you are copying and pasting large amounts its not the same as opening the file. This is made worse if the encoding of the material on the clipboard needs to be changed as thats another copy again.
None of the above is a surprise, exceeding memory is not a bug or issue we can do anything about.
@hdaz000700 What is interesting is the confused display shown in your screenshots where the end of line characters are not at the ends of the lines. That indicates that something else is going on as well.
Thats what I'm trying to investigate, as it may indicate a bug that may be making your problem worse.
So for the moment all I'm interested in is if the file opens correctly in Geany and Scite, Linux and Windows. Then we can look for causes if the file opens ok, taking it one step at a time to prevent getting confused again.
It would be useful to know that no testers are Reloading the document while `Prefs->Various->keep_edit_history_on_reload` is enabled. This will absolutely destroy RAM for ridiculously large files.
Hi Elextr,
The file opens fine within both Linux and Windows from both Geany and Scite.
Within Windows its not until I start doing anything like entering text things lock up with the LF down the side maybe I have been too impatient to see if it ever unlocks from this position but as far as I recall I have left it in this state for quite some time and nothing has unlocked, also nothing else on the laptop shows any signs of swapping locking up.
Do you really consider 250 MB - 300 MB as a ridiculously large file for 2017? I agree its a little larger than your average but the convenience I get keeping it as one file outweighs the pain it would be to maintain multiple files.
At this moment in time I have invested too much time into Geany to switch.. :-o I use vi / vim / gvim all day long its not the same.... my mind is set to work in a totally different work flow... shudders...
I honestly cant recall if I tested this on windows or linux but I have tested the size of the file just in case there were going to be issues down the road, about a year a go and things worked well then with over 600 MB of file... << older version of Geany probably version 1.25 as its still in my windows download folder date stamp 08/09/2015 so maybe a little over a year ago >>
Regards Darren
Do you really consider 250 MB - 300 MB as a ridiculously large file for 2017?
For a source code file, it's very large. Even auto-generated source code would rarely (if ever) get that large. Not that it's a use case that shouldn't be supported if possible. If the machine can handle it and you're patient enough, then the only limit should be the size of a signed integer (usually 2^32) minus one.
@codebrainz keep_edit_history_on_reload is currently ticked.
If you're Reloading the file (either by the menu, toolbar or <kbd>Ctrl</kbd>+<kbd>r</kbd>), then each time it will duplicate the entire file contents into memory. For example if you reload a 300MB file, it will put all 300MB into the undo buffer, plus whatever the new contents are will be resident in the buffer. If you reload it a second time, the undo buffer will contain 600MB of data, and so on. That's why I mentioned it.
also nothing else on the laptop shows any signs of swapping locking up.
Ok, swapping is probably not happening since it will make everything slow.
Within Windows its not until I start doing anything like entering text things lock up with the LF down the side.
Ok, so Geany hangs, but in a post above you say Scite says it has run out of memory and closes, is that what happens when you do the same (or reasonably similar) things with Scite?
Because if Scite is saying Scintilla is out of memory, then its likely that Scintilla in Geany is too, but Geany misses the error and just gets in an infinite loop instead. That should be fixed if we can find it. However there are not many contributors to Geany who use Windows, which is why I am continuing to ask your assistance. (I don't even have windows in the house)
Do you really consider 250 MB - 300 MB as a ridiculously large file for 2017?
There should be no hard limit in Geany, but as I said above, if its running out of memory then thats the limit, nothing we can do. So can you please be more specific about what you did so we can try to determine how much memory the things you do might take.
Note its not always obvious what it takes. For example when you are editing and you increase the buffer contents beyond its current memory size the buffer memory must be increased in size, which means temporarily there are two copies in memory, as a new bigger buffer is allocated, and the contents of the old one copied to the new one, before the old one is de-allocated. And remember the buffer is already twice the size of the file since it has a second byte for each byte in the file for storing the styling information. So adding one character can cause the memory usage to be temporarily four times the file size, or 1.2GB for the 300MB file, which is a significant part of the 4GB that a 32 bit application can address, just for the buffer, without all the supporting structures. I don't know how much data memory 32 bit Windows allows, maybe its only 2GB.
This happens inside Scintilla and is potentially Scite's "out of memory" message, so if you provide details of what you did to crash Scite/hang Geany we can look at its potential usage. Geany on Linux is a 64 bit program, so it will not run out of address space before real memory, and swapping kicks in which "hangs" the whole machine.
As @codebrainz said, Scintilla and Geany's use-case is for editing human source files, so its code is designed on that premise. If it was designed specifically for large files it would probably be designed differently.
I don't know how much data memory 32 bit Windows allows, maybe its only 2GB
2GB for 32-bit processes running on 32 or 64-bit Windows (ex. Geany), 8TB for 64-bit processes on 64-bit Windows, by default. At least according to [their docs](https://msdn.microsoft.com/en-us/library/aa366778.aspx#memory_limits)
@codebrainz thanks, so if all the rest of data memory (the per line data and everything else) adds up to say 500-600MB then it will be exceeded on buffer resize for a file near 300-400MB. So it looks more and more like the problem, but it would be good if Geany didn't just hang :)
Hi Guys,
Just focusing on entering text rather than copy n paste.
So I create a file like the following and move it to windows :
`#!/bin/bash filecount=0 while [[ $filecount -lt 1 ]] ; do filesize=$RANDOM filesize=$(($filesize+300000000)) yes "111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111 111111111111111111111111111111111111111111111111111111111111" | head -c $filesize" > /home/daz/file_${filecount}.$RANDOM ((filecount++)) done`
I then within Geany type a number of "1's" 7 to 10 times and everything locks up.
Doing the same process within SciTE does lock SciTE up.
Regards Darren
@hdaz000700 see https://github.com/geany/geany/pull/1572
@elextr
Is there or will there be any advance to have a 64bit Windows version of Geany ?
Regards Daz
When somebody contributes the changes neccessary for the build to work, and makes Travis use it and the nightly builds and the release builds then yeah. I don't know if anybody has even tried with msys 64 bit.
github-comments@lists.geany.org