FWIW, when GLib aborts the process on malloc fail, it writes this to the debug messages (if `-v` is used):
GLib: .../gmem.c:165: failed to allocate XXX bytes
I just tested opening an 18GB file on my Win7 machine that has 16GB of RAM. This is when opening a file, so presumably when Geany is reading it into a string before it gets to Scintilla.