OK, I have a fix for making a11y quite a lot faster in this case, see #2097. It's still "slow", but it's now a matter of seconds, not tens of minutes or more.
It still might be good to only call all this when it's actually useful, although that'd mean a lot less testing -- and issues like that wouldn't be noticed as much.