The colours are not changing, only the background. This is to indicate that this is no longer the primary X selection (the thing that gets pasted by middle mouse click) but it is still the selection for window specific actions when this is the current window, eg it gets copied to the clipboard on `ctrl-c`.
IIRC there is not a setting to change this value, there was a PR but it has not been updated to comments so its not merged yet.