Actually on Windows, Geany is binary relocatable. So what the OP requests should be possible in theory.
But I don't think it's necessary to add anything to Geany itself. @eadmaster I suggest to test just putting the Geany installation directory to an USB drive or whatever medium you prefer and add a little batch file in the directory to execute "bin/geany.exe --config=portable_config" or something like this. Would this be suffice?