The only way then would be to change the width of that margin depending on which lines are shown, which is likely to be annoying behavior when near a boundary (e.g. going up/down a line could shift the whole display to the left).
IMO this is at most a feature request, and I'd be tempted to close as WONTFIX. But maybe someone else would find this important enough to come up with a patch or a plugin that changes that (if it's a core patch, if would likely need a preference).