There is no limit on the number of files open in an instance of Geany unless somehow Windows is limiting it, there are users who have hundreds of files open.
Sounds more like the interprocess communication in Windows is gumming up, but with no information about exactly what you are doing, the versions of Geany, Windows, etc can't help you and nobody can replicate the problem.